Using virtual networking devices and routing information to associate network addresses with computing nodes

ABSTRACT

Techniques are described for providing managed virtual computer networks that have a configured logical network topology with virtual networking devices, such as by a network-accessible configurable network service, with corresponding networking functionality provided for communications between multiple computing nodes of the virtual computer network by emulating functionality that would be provided by the virtual networking devices if they were physically present. In some situations, the networking functionality provided for a managed computer network of a client includes receiving routing communications directed to the virtual networking devices and using included routing information to update the configuration of the managed computer network, such as to allow at least some computing nodes of a managed computer network to dynamically signal particular types of uses of one or more indicated target network addresses and/or to dynamically signal use of particular external public network addresses based on such routing information.

BACKGROUND

Many companies and other organizations operate computer networks thatinterconnect numerous computing systems to support their operations,with the computing systems alternatively co-located (e.g., as part of aprivate local area network, or “LAN”) or instead located in multipledistinct geographical locations (e.g., connected via one or more otherprivate or shared intermediate networks). For example, data centershousing significant numbers of interconnected computing systems havebecome commonplace, such as private data centers that are operated byand on behalf of a single organization, as well as public data centersthat are operated by entities as businesses. Some public data centeroperators provide network access, power, and secure installationfacilities for hardware owned by various customers, while other publicdata center operators provide “full service” facilities that alsoinclude hardware resources made available for use by their customers.However, as the scale and scope of typical data centers and computernetworks has increased, the task of provisioning, administering, andmanaging the associated physical computing resources has becomeincreasingly complicated.

The advent of virtualization technologies for commodity hardware hasprovided some benefits with respect to managing large-scale computingresources for many customers with diverse needs, allowing variouscomputing resources to be efficiently and securely shared betweenmultiple customers. For example, virtualization technologies such asthose provided by VMWare, XEN, Linux's KVM (“Kernel-based VirtualMachine”), or User-Mode Linux may allow a single physical computingmachine to be shared among multiple users by providing each user withone or more virtual machines hosted by the single physical computingmachine, with each such virtual machine being a software simulationacting as a distinct logical computing system that provides users withthe illusion that they are the sole operators and administrators of agiven hardware computing resource, while also providing applicationisolation and security among the various virtual machines.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1B are network diagrams illustrating example embodiments ofconfiguring and managing networking functionality provided for computingnodes belonging to a managed computer network.

FIGS. 2A-2E illustrate examples of managing communications betweencomputing nodes of a managed virtual overlay computer network.

FIG. 3 is a block diagram illustrating example computing systemssuitable for executing an embodiment of a system for managingcommunications between computing nodes.

FIG. 4 illustrates a flow diagram of an example embodiment of a CNSSystem Manager routine.

FIGS. 5A-5B illustrate a flow diagram of an example embodiment of a CNSCommunication Manager routine.

FIGS. 6A-6B illustrate a flow diagram of an example embodiment of a CNSNetwork Routing Manager routine.

DETAILED DESCRIPTION

Techniques are described for providing virtual networking functionalityfor managed computer networks, such as for computer networks that aremanaged for and provided on behalf of users or other entities. Suchmanaged computer networks may in some embodiments be provided by aconfigurable network service to users or other entities who arecustomers (e.g., for a fee) or otherwise clients of the configurablenetwork service, such as to remote clients that access the configurablenetwork service and/or the provided managed computer networks fromremote locations over one or more intervening networks. In at least someembodiments, the described techniques enable a user to configure orotherwise specify a network topology for a managed computer networkbeing provided for the user, such as to separate multiple computingnodes of the managed computer network into multiple logical sub-networksinterconnected by one or more specified networking devices, or tootherwise include one or more specified networking devices that are eachassociated with a specified group of the multiple computing nodes. Aftera network topology is specified for a managed computer network,networking functionality corresponding to the specified network topologymay be provided in various manners, such as by implementing the managedcomputer network as a virtual computer network overlaid on one or moreother computer networks, and by providing functionality corresponding tothe specified network topology without physically implementing at leastsome of the specified network topology. For example, the configurablenetwork service may in some embodiments handle communications betweencomputing nodes of a managed computer network in accordance with itsspecified network topology by emulating at least some types offunctionality that would be provided by virtual networking devices forthe managed computer network if they were physically present, butwithout physically providing those networking devices. Similarly, insome embodiments, routing communications that include routinginformation for the managed computer network and that are directed tothe specified networking devices may be managed without physicallyproviding the networking devices, such as by intercepting the routingcommunications and using the routing information to update the networktopology for the managed computer network. In at least some embodiments,some or all of the described techniques are automatically performed byembodiments of a Network Routing Manager (“NRM”) module and/or one ormore other modules, such as one or more NRM modules and multiplecommunication manager modules that are part of a network-accessibleconfigurable network service that provides configurable computernetworks to clients.

Thus, in at least some embodiments, a configurable network servicesupports networking devices that are not physically provided for amanaged computer network, such as virtual networking devices that arepart of a logical network topology for the managed computer network,with the configurable network service emulating or otherwise providingat least some of the functionality of such virtual networking devices.For example, the configurable network service may perform variousactions in at least some such embodiments to enable the virtualnetworking devices to participate in routing protocols in the samemanner or a similar manner as would an actual physically providednetworking device of the managed computer network. Such routingprotocols generally enable physically provided network routers and othernetworking devices of a computer network, as well as other computingnodes that provide functionality to facilitate handling communicationsfor the computer network, to communicate with each other and exchangevarious types of routing information, so that network topology andoptionally network operation characteristics may be shared and usedthroughout the computer network. For example, some routing protocolsdetermine best paths to destinations based on the minimum number of hopsor on some other minimum distance measure, and exchange routinginformation that may include routing tables indicating a total distancerouting “cost” and next best hop for each known destination—thus, anetwork router that receives such routing table information may updateits own routing table if the received routing information includes any“better” (i.e., lower cost) routes to known destinations, as well as toadd information for any previously unknown destinations. Other routingprotocols determine best paths to destinations based on various routingcost measures that are not limited to minimum distance, such as toinstead or also consider information corresponding to networktransmission characteristics such as actual network bandwidth, latency,reliability, load, etc.—such other routing protocols may exchangerouting information that includes connectivity-related information(e.g., who are a sending router's nearest neighbors) and optionallynetwork transmission characteristics information. A network router thatreceives such connectivity-related information may use it to optionallyupdate its map of the connectivity of the various routers in thenetwork, which it may then use to calculate the current best paths fromitself to the various destinations. The types of destinations that arouting protocol may represent when using such routing cost informationor other information may include, for example, a range of IP addresses(e.g., IPv4 addresses, IPv6 addresses, etc.) that are represented by anIP address prefix, or instead one or more network addresses that arerepresented in another form (e.g., using MPLS labels). Typically, eachrouter will generate a subset of the set of total routing information(also referred to as a “routing table” and/or “Routing InformationBase,” or “RIB”) that it has obtained, with the generated subset(referred to as a “forwarding table” and/or “Forwarding InformationBase,” or “FIB”) including only the preferred routes (e.g., one routefor each IP address prefix), as determined based on the routing protocolin use and available information. Each router's generated FIB is used torapidly make forwarding decisions for network communication packets thatare received. In addition, in some situations, a router may generate andstore multiple alternative routes for a destination as part of its FIBinformation, such as to represent ECMP (“Equal Cost Multi-Path”) routinginformation that is received—if so, the router may use varioustechniques to select a particular one of the routes when forwarding anetwork packet to that destination. Thus, each physically providednetwork router will typically make its own decisions regarding how toroute network communication packets in an attempt to maximize the localoperation of the network near the router, using whatever information islocally available to the router.

As discussed in greater detail below, a managed computer network may beconfigured and provided in various manners, including in someembodiments by implementing a managed computer network as a virtualcomputer network that is overlaid on a distinct substrate network, witheach of the multiple computing nodes of the managed computer networkbeing initially configured to use at least one of multiple virtualnetwork addresses for the managed computer network. In addition, such amanaged computer network may be configured to include one or morespecified networking devices, which may be implemented as virtualnetworking devices that are not physically provided, as noted above.Furthermore, a managed computer network may be configured to include twoor more computing nodes that are alternatives for at least somecommunications, such as two or more alternative computing nodes that maybe used as intermediate computing nodes along multiple alternativerouting paths from one or more particular source computing nodes to oneor more particular destination computing nodes or other destinations.Such multiple alternative computing nodes may include, as illustrativeexamples, multiple alternative intermediate computing nodes that providesoftware-based VPN connection endpoints for multiple VPN connections viawhich a remote external computer network is reachable, multiplealternative intermediate software-based firewall computing nodes betweendifferent logical subnets of the managed computer network (e.g., with aprimary firewall, and one or more alternative backup firewalls that areavailable if the primary firewall becomes unavailable), etc. Moreover,in at least some embodiments, one or more of the multiple virtualnetwork addresses used by a managed computer network may further beconfigured to be a target network address that has a particularspecified type of use within the managed computer network, such as by aclient for whom the managed computer network is provided.

In some embodiments, the providing of networking functionality for amanaged computer network provided by a configurable network service to aclient includes using routing information in routing communicationsreceived from computing nodes of the managed computer network in variousmanners, including by allowing at least some computing nodes of themanaged computer network to dynamically signal particular types of usesof one or more indicated target network addresses based on such routinginformation. For example, the configurable network service may in someembodiments intercept routing communications that are directed tovirtual networking devices of the managed computer network (e.g., fromclient-controlled computing nodes of the managed computer network thatfacilitate handling communications for the managed computer network),and use at least some of the included routing information to dynamicallyconfigure the network topology of the managed computer network. The useof the included routing information may include, for example, analyzingthe routing information to determine if any computing nodes of themanaged computer network have dynamically indicated use of particulartarget network addresses, and if so performing correspondingconfiguration for the managed computer network. For example, aparticular computing node may dynamically announce that it is newlyassociated with a particular target virtual network address by sending arouting communication that announces that the target virtual networkaddress is available via that computing node, so that communicationsdirected to that target virtual network address will be routed to thatcomputing node. In other embodiments and situations, a computing nodethat participates in a routing protocol for the managed computer networkmay instead dynamically announce the availability of a target networkaddress via that computing node, but with the computing node acting asan intermediate destination that will forward communications directed tothat target network address on to one or more other computing nodes thatuse the target network address.

Target network addresses may be associated with various types of uses invarious embodiments. For example, a particular target virtual networkaddress for a managed computer network may be specified (e.g., by theclient for whom the managed computer network is provided) to representan anycast address having an associated group of one or more alternativecomputing nodes, and a particular computing node of the managed computernetwork may dynamically indicate that it is a part of that anycast groupby sending routing information that announces availability of thattarget virtual network address for that computing node. As anotherexample, a particular computing node of the managed computer network maydynamically indicate via sent routing information that it providesavailability to a particular target virtual network address, such as ifthat target virtual network address has dynamically been migrated tothat computing node from another computing node that previously usedthat target virtual network address (e.g., for another computing nodethat has become unavailable, such that the particular computing node istaking over use of that target virtual network address). Additionaldetails related to configuring target network addresses and to usingrouting information to specify relationships of computing nodes withtarget network addresses are included below.

In addition, in some embodiments one or more computing nodes of amanaged computer network may supply routing information that indicatesone or more external public network addresses to be associated with themanaged computer network, such as a range of public network addresses(e.g., an IP address prefix) that are globally routable over theInternet to enable external computer systems to communicate with themanaged computer network. As one example, an embodiment of theconfigurable network service may use a particular data center to provideone or more managed computer networks, including by using virtualnetwork addresses that are private for the configurable network serviceand data center, but with a particular managed computer network havingone or more associated external public network addresses to enablecomputer systems outside the configurable network service and datacenter to send communications to that managed computer network. When theconfigurable network service receives routing information that indicatessuch specified external public network addresses, it may optionallyperform various activities to verify the availability andappropriateness of the use of those external public network addresses(e.g., that a client to whom the managed computer network is provided isauthorized to use those public network addresses), and then publiclyannounce or advertise the availability of those external public networkaddress(es) (e.g., by publicly announcing externally to the data centerthat those external public network addresses are mapped to computersystems at that data center). As one example, the configurable networkservice may use the Border Gateway Protocol (“BGP”) to publicly announcethe availability of such external public network addresses for a managedcomputer network provided for a client, such as under control of theconfigurable network service or by supplying the external public networkaddress information to remote facilities of the client that areconfigured to perform the public announcement. Additional detailsrelated to the specification and use of such public network addressesare included below.

As noted above, the described techniques enable a user or other entityto in at least some embodiments configure or otherwise specify one ormore virtual networking devices for a managed computer network beingprovided on behalf of the user or entity, and include performing variousautomated actions to support such specified virtual networking devices(e.g., intercepting routing communications directed to virtualnetworking devices, and using routing information in the routingcommunications in various manners), including in embodiments in whichthe managed computer network is a virtual computer network. Beforediscussing some details of providing virtual networking functionalitycorresponding to such specified virtual networking devices for a managedcomputer network, some aspects of such managed computer networks in atleast some embodiments are introduced.

In particular, a managed computer network between multiple computingnodes may be provided in various ways in various embodiments, such as inthe form of a virtual computer network that is created as an overlaynetwork using one or more intermediate physical networks that separatethe multiple computing nodes. In such embodiments, the intermediatephysical network(s) may be used as a substrate network on which theoverlay virtual computer network is provided, with messages and othercommunications between computing nodes of the overlay virtual computernetwork being passed over the intermediate physical network(s), but withthe computing nodes being unaware of the existence and use of theintermediate physical network(s) in at least some such embodiments. Forexample, the multiple computing nodes may each have a distinct physicalsubstrate network address that corresponds to a location of thecomputing node within the intermediate physical network(s), such as asubstrate IP (“Internet Protocol”) network address (e.g., an IP networkaddress that is specified in accordance with IPv4, or “Internet Protocolversion 4,” or in accordance with IPv6, or “Internet Protocol version6,” such as to reflect the networking protocol used by the intermediatephysical networks). In other embodiments, a substrate network on which avirtual computer network is overlaid may itself include or be composedof one or more other virtual computer networks, such as other virtualcomputer networks implemented by one or more third parties (e.g., by anoperator or provider of Internet or telecom infrastructure).

When computing nodes are selected to participate in a managed computernetwork that is a virtual computer network overlaid on a substratenetwork, each computing node of the managed virtual computer network mayalso be assigned one or more virtual network addresses for the virtualcomputer network that are unrelated to those computing nodes' substratenetwork addresses, such as from a range of virtual network addressesused for the managed virtual computer network—in at least someembodiments and situations, the managed virtual computer network beingprovided may further use a networking protocol that is different fromthe networking protocol used by the substrate network (e.g., with thevirtual computer network using the IPv4 networking protocol, and thesubstrate computer network using the IPv6 networking protocol). Thecomputing nodes of the virtual computer network inter-communicate usingthe virtual network addresses (e.g., by sending a communication toanother destination computing node by specifying that destinationcomputing node's virtual network address as the destination networkaddress for the communication), but the substrate network may beconfigured to route or otherwise forward communications based onsubstrate network addresses (e.g., by physical network router devicesand other physical networking devices of the substrate network). If so,the overlay virtual computer network may be implemented from the edge ofthe intermediate physical network(s), by modifying the communicationsthat enter the intermediate physical network(s) to use substrate networkaddresses that are based on the networking protocol of the substratenetwork, and by modifying the communications that leave the intermediatephysical network(s) to use virtual network addresses that are based onthe networking protocol of the virtual computer network. Additionaldetails related to the provision of such an overlay virtual computernetwork are included below.

In at least some embodiments, a network-accessible configurable networkservice (“CNS”) is available for use by customers, such as a CNSprovided by a corresponding CNS system that provides and manages overlayvirtual computer networks for remote customers (e.g., users and otherentities). Such a CNS service may, for example, provide and use numerouscomputing nodes that are in one or more geographical locations (e.g., inone or more data centers) and that are inter-connected via one or moreintermediate physical networks or other connections. The CNS system mayuse various communication manager modules at the edge of the one or moreintermediate physical networks to manage communications for the variousoverlay virtual computer networks as they enter and leave theintermediate physical network(s), and may use one or more system managermodules to coordinate other operations of the CNS system. For example,to enable the communication manager modules to manage communications forthe overlay virtual computer networks being provided, the CNS system maytrack and use various information about the computing nodes of eachvirtual computer network being managed, such as to map the substratephysical network address of each such computing node to the one or moreoverlay virtual network addresses associated with the computing node.Such mapping and other information may be stored and propagated invarious manners in various embodiments, including centrally or in adistributed manner, as discussed in greater detail below.

Furthermore, in order to provide managed virtual computer networks tousers and other entities, the CNS system allows users and other entitiesto interact with the CNS system in at least some embodiments toconfigure a variety of types of information for virtual computernetworks that are provided by the CNS system on behalf of the users orother entities, and may track and use such configuration information aspart of providing those virtual computer networks. The configurationinformation for a particular managed virtual computer network havingmultiple computing nodes may include, for example, one or more of thefollowing non-exclusive list: a quantity of the multiple computing nodesto include as part of the virtual computer network; one or moreparticular computing nodes to include as part of the virtual computernetwork; a range or other group of multiple virtual network addresses toassociate with the multiple computing nodes of the virtual computernetwork; particular virtual network addresses to associate withparticular computing nodes or particular groups of related computingnodes; a type of at least some of the multiple computing nodes of thevirtual computer network, such as to reflect quantities and/or types ofcomputing resources to be included with or otherwise available to thecomputing nodes; a geographic location at which some or all of thecomputing nodes of the virtual computer network are to be located; etc.In addition, the configuration information for a virtual computernetwork may be specified by a user or other entity in various manners invarious embodiments, such as by an executing program of the user orother entity that interacts with an API (“application programminginterface”) provided by the CNS system for that purpose and/or by a userthat interactively uses a GUI (“graphical user interface”) provided bythe CNS system for that purpose.

FIG. 1A is a network diagram illustrating an example of anetwork-accessible service that provides client-configurable managedcomputer networks to clients. In particular, in this example, at leastsome of the managed computer networks may be virtual computer networks,such as virtual computer networks that are created and configured asnetwork extensions to existing remote private computer networks ofclients, although in other embodiments the managed computer networks mayhave other forms and/or be provided in other manners. After configuringsuch a managed computer network being provided by the network-accessibleservice, a user or other client of the network-accessible service mayinteract from one or more remote locations with the provided computernetwork, such as to execute programs on the computing nodes of theprovided computer network, to dynamically modify the provided computernetwork while it is in use, etc. In particular, in the illustratedexample of FIG. 1A, a configurable network service (“CNS”) 105 isavailable that provides functionality to clients (not shown) over one ormore public networks 100 (e.g., over the Internet) to enable the clientsto access and use managed computer networks provided to the clients bythe CNS 105, including to enable the remote clients to dynamicallymodify and extend the capabilities of their remote existing privatecomputer networks using cloud computing techniques over the publicnetwork 100. In addition, the CNS 105 may provide functionality toenable clients to specify logical network topology information for theirmanaged virtual computer networks, and if so provides correspondingnetworking functionality (e.g., under control of an NRM module of theCNS 105, such as one of the other modules 115), as described in greaterdetail later.

In the example of FIG. 1A, a number of clients interact over the publicnetwork 100 with a system manager module 110 of the CNS 105 to createand configure various managed computer networks 120 being provided bythe CNS 105, with at least some of the provided computer networks 120being private computer network extensions to remote existing clientprivate networks 130, and with at least some such of the providedcomputer network extensions 120 being configured to enable privateaccess from one or more corresponding client private networks 130 overthe public network 100 (e.g., via VPN connections established overinterconnections 100 a and 100 b, or via other types of private ornon-private interconnections). In this example embodiment, the managermodule 110 assists in providing functionality of the CNS 105 to theremote clients, such as in conjunction with various optional othermodules 115 of the CNS 105 and optionally various computing nodes 125and/or networking devices (not shown) of a substrate network that areused by the CNS 105 to provide the managed computer networks 120. In atleast some embodiments, the CNS manager module 110 may execute on one ormore computing systems (not shown) of the CNS 105, and may provide oneor more APIs that enable remote computing systems to programmaticallyinteract with the module 110 to access some or all functionality of theCNS 105 on behalf of clients (e.g., to create, configure, and/orinitiate use of managed computer networks 120). In addition, in at leastsome embodiments, clients may instead manually interact with the module110 (e.g., via a GUI provided by the module 110) to perform some or allsuch actions.

The public network 100 in FIG. 1A may be, for example, a publiclyaccessible network of linked networks, possibly operated by distinctparties, such as the Internet. The remote client private networks 130may each include one or more existing private networks, such as acorporate or other private network (e.g., home, university, etc.) thatis partially or wholly inaccessible to non-privileged users, and thatincludes computing systems and/or other networked devices (not shown) ofa client. In the illustrated example, the provided computer networks 120each include multiple computing nodes (not shown), at least some ofwhich may be from a plurality of optional computing nodes 125 providedby or otherwise under the control of the CNS 105, while in otherembodiments at least some other computing systems 135 may be used toprovide some or all computing nodes for one or more of the providedcomputer networks 120, such as other computing systems 135 that areprovided by or under control of the client for whom a computer network120 that uses those other computing systems 135 is provided, or othercomputing systems 135 that are provided by third parties (e.g., for afee). Each of the provided computer networks 120 may be configured invarious ways by the clients for whom they are provided, such as to be anextension to a corresponding remote computer network 130, and may eachbe a private computer network that is accessible only by the client thatcreates it, although in other embodiments at least some computernetworks provided by the CNS 105 for clients may be publicly accessibleand/or may be standalone computer networks that are not extensions toother existing computer networks 130. Similarly, while at least some ofthe provided computer networks 120 in the example may be extensions toremote client computer networks 130 that are private networks, in otherembodiments the provided computer networks 120 may be extensions toother client computer networks 130 that are not private networks.

Private access between a remote client private computer network 130 andcorresponding private computer network extension 120 provided for aclient may be enabled in various ways, such as by establishing a VPNconnection or other private connection between them that allowsintercommunication over the public network 100 in a private manner. Forexample, the CNS 105 may automatically perform appropriate configurationon its computing nodes and other computing systems to enable VPN accessto a particular private network extension 120 of a client, such as byautomatically configuring one or more VPN mechanisms hosted by the CNS105 (e.g., software and/or hardware VPN mechanisms), and/or mayautomatically provide appropriate configuration information to theclient (e.g., credentials, access points, and/or other parameters) toallow a VPN mechanism hosted on the remote client private network 130(e.g., a software VPN endpoint that is provided by one of the multiplecomputing nodes of the provided network extension 120) to establish theVPN access. After VPN access has been appropriately enabled and/orconfigured, a VPN connection may be established between the remoteclient private network and the provided private network extension, suchas initiated by the client using IPsec (“Internet Protocol Security”) orother appropriate communication technologies. For example, in someembodiments, a VPN connection or other private connection may beestablished to or between networks that use MPLS (“Multi Protocol LabelSwitching”) for data transmission, such as instead of an IPsec-based VPNconnection. In addition, in the illustrated example, variousnetwork-accessible remote resource services 140 may optionally beavailable to remote computing systems over the public network 100,including to computing nodes on the remote client private networks 130.The resource services 140 may provide various functionality to theremote computing nodes, such as for at least some of the resourceservices 140 to provide remote computing nodes with access to varioustypes of network-accessible computing-related resources. Furthermore, atleast some of the computer networks 120 that are provided by the CNS 105may be configured to provide access to at least some of the remoteresource services 140, with that provided access optionally appearing tocomputing nodes of the provided computer networks 120 as being locallyprovided via virtual connections 117 that are part of the providedcomputer networks 120, although the actual communications with theremote resource services 140 may occur over the public networks 100(e.g., via interconnections 100 b and 100 c). In addition, in at leastsome embodiments, multiple distinct provided computer networks 120 maybe configured to enable inter-access with each other.

The provided computer networks 120 may each be configured by clients invarious manners. For example, in at least some embodiments, the CNS 105provides various computing nodes 125 that are available for use withcomputer networks provided to clients, such that each provided computernetwork 120 may include a client-configured quantity of multiple suchcomputing nodes that are dedicated for use as part of that providedcomputer network. In particular, a client may interact with the module110 to configure a quantity of computing nodes to initially be includedin a computer network provided for the client (e.g., via one or moreprogrammatic interactions with an API provided by the CNS 105). Inaddition, the CNS 105 may provide multiple different types of computingnodes in at least some embodiments, such as, for example, computingnodes with various performance characteristics (e.g., processor speed,memory available, storage available, etc.) and/or other capabilities. Ifso, in at least some such embodiments, a client may specify the types ofcomputing nodes to be included in a provided computer network for theclient. In addition, in at least some embodiments, a client may interactwith the module 110 to configure network addresses for a computernetwork provided for the client (e.g., via one or more programmaticinteractions with an API provided by the CNS 105), and network addressesmay later be dynamically added, removed or modified for a providedcomputer network of a client in at least some such embodiments, such asafter the provided computer network has already been in use by theclient. In addition, in at least some embodiments, a client may interactwith the module 110 to configure network topology information for acomputer network provided for the client (e.g., via one or moreprogrammatic interactions with an API provided by the CNS 105), and suchnetwork topology information may later be dynamically modified for aprovided computer network in at least some such embodiments, such asafter the provided computer network has already been in use by theclient. Furthermore, in at least some embodiments, a client may interactwith the module 110 to configure various network access constraintinformation for a computer network provided for the client (e.g., viaone or more programmatic interactions with an API provided by the CNS105), and such network access constraint information may later bedynamically modified for a provided computer network in at least somesuch embodiments, such as after the provided computer network hasalready been in use by the client.

Network addresses may be configured for a provided computer network invarious manners in various embodiments. For example, if a particularprovided computer network that is being configured is an extension to anexisting remote client computer network, the client may specify one ormore address ranges (e.g., a Classless Inter-Domain Routing (“CIDR”)address block) or other groups of network addresses for the providedcomputer network that are a subset of the network addresses used by theexisting remote client computer network, such that at least some of thespecified network addresses are used for the computing nodes of theprovided computer network. Such configured network addresses may in somesituations be virtual or private network addresses that are not directlyaddressable from computing systems on the public network 100 (e.g., ifthe existing remote client computer network and the correspondingprovided network extension use network address translation techniquesand/or virtual networking techniques for the client computer network andits provided network extension), while in other situations at least someof the configured network addresses may be public network addresses thatare directly addressable and globally routable from computing systems onthe public network 100 (e.g., a public network address that is a staticInternet-routable IP address or other non-changing network address). Inother embodiments, the CNS 105 may automatically select networkaddresses to be used for at least some computing nodes of at least someprovided computer networks, such as based on network addresses that areavailable for use by the CNS 105, based on selecting network addressesthat are related to network addresses used by remote existing computernetworks corresponding to the provided computer networks, etc.Furthermore, if two or more of the computer networks provided by the CNS105 are configured to enable inter-communications between the providedcomputer networks (e.g., for two or more computer networks provided to asingle customer, such as for different departments or groups within asingle organization; for two or more computer networks provided to twoor more distinct customers; etc.), the CNS 105 may in some embodimentsautomatically select network addresses to be used for at least somecomputing nodes of those provided computer networks to facilitate theinter-communications, such as by using different network addresses forthe various provided computer networks. In addition, in at least someembodiments in which the CNS 105 provides virtual networks to clients,such as by using overlay networks on a substrate network, each clientmay be allowed to specify any network addresses to be used for theirprovided computer networks, even if multiple clients specify the same oroverlapping network addresses for their respective provided computernetworks—in such embodiments, the CNS 105 manages the network addressesdistinctly for each client, such that a first client may have a firstcomputing node associated with a particular specified network addressfor the first client's provided computer network, while a distinctsecond client may have a distinct second computing node associated withthe same particular specified network address for the second client'sprovided computer network. Once network addresses are configured orotherwise determined for a provided computer network, the CNS 105 mayassign the network addresses to various of the computing nodes selectedfor the provided computer network, such as in a random manner, by usingDHCP (“Dynamic Host Configuration Protocol”) or other techniques fordynamic assignment of network addresses, etc. Moreover, in at least someembodiments, one or more particular virtual network addresses associatedwith a managed computer network may be configured to be target networkaddresses that each has a particular specified type of use within themanaged computer network.

Network topology information may be configured for a provided computernetwork in various manners in various embodiments. For example, a clientmay specify particular types of networking devices (e.g., routers,switches, etc.) and/or other network appliance devices or nodes (e.g.,load balancers, firewalls, proxies, network storage devices, printers,etc.) to be part of the provided computer network, and may specifyinterconnectivity information between networking devices and computingnodes. Furthermore, in at least some embodiments, the CNS 105 mayprovide available computing nodes in multiple geographical locations(e.g., in multiple geographically distributed data centers), and theconfiguration information specified by a client for a provided computernetwork may further indicate one or more geographical locations in whichcomputing nodes of the provided computer network are to be located(e.g., to provide fault tolerance among the computing nodes of aprovided computer network by having them located in multiplegeographical locations), and/or may otherwise provide information aboutpreferences or requirements of how the computing nodes of the providedcomputer network are to interoperate that is used by the CNS 105 toselect one or more such geographical locations (e.g., minimum or maximumnetwork latency or bandwidth for computing node intercommunications;minimum or maximum network proximity between computing nodes; minimum ormaximum geographic proximity between computing nodes; having localaccess to particular resources or functionality that is not available inall such geographic locations; having specified locations relative toother external computing systems, such as to a remote computer networkof the client and/or to a remote resource service; constraints or otherpreferences based on the cost of obtaining use of particular computingnodes and/or for particular types of interactions with particularcomputing nodes, such as costs associated with providing data to and/orfrom those computing nodes; etc.). As discussed in greater detailelsewhere, in at least some embodiments, the interconnections andintercommunications between computing nodes of a provided computernetwork are managed using an underlying substrate network of the CNS105, and if so, some or all of the configured network topologyinformation may be simulated or otherwise emulated in at least some suchembodiments using the underlying substrate network and correspondingmodules of the CNS 105. For example, each of the computing nodesprovided by the CNS 105 may be associated with a node communicationmanager module of the CNS 105 that manages communications to and fromits associated computing node(s), and if so, the associatedcommunication manager module for a computing node may take variousactions to emulate desired functionality of a network with respect tothat computing node, as discussed in greater detail elsewhere.

Network access constraint information may also be configured for aprovided computer network in various manners in various embodiments. Forexample, a client may specify information about whether and how some orall of the computing nodes of a provided computer network are allowed tocommunicate with other computing nodes of the provided computer networkand/or with other external computing systems, such as based on one ormore of the following: directions of communications (incoming versusoutgoing); types of communications (e.g., based on the types of contentincluded and/or the types of communication protocols used, such as toallow HTTP requests for text but not images and to not allow FTPrequests); locations of other computing systems (e.g., whether part ofthe provided computer network, part of a remote client computer networkcorresponding to the provided computer network, part of a remoteresource service to which access has been established, external to theprovided computer network and any corresponding remote client computernetwork, etc.); types of other computing systems; etc. In a mannersimilar to that for network topology information, the CNS 105 mayenforce network access constraint information for provided computernetworks in various manners.

Thus, managed computer networks may be provided for clients in variousmanners in various embodiments, and may be configured to have varioustypes of functionality in various embodiments.

In addition, in at least some embodiments, the computing nodes of themanaged computer networks may be physical computing systems and/or maybe virtual machines that are each hosted on one or more physicalcomputing systems, and the communications that are handled for managedcomputer networks may include transmissions of data (e.g., messages,packets, frames, streams, etc.) in various formats. As previously noted,some or all computing nodes used for a particular provided overlayvirtual computer network may in some embodiments be provided by the CNSsystem for use by users, while in other embodiments some or all suchcomputing nodes may instead be provided by a user who uses thosecomputing nodes. Furthermore, in at least some situations, an embodimentof the CNS system may be part of or otherwise affiliated with a programexecution service (or “PES”) that executes multiple programs on behalfof multiple customers or other users of the service, such as a programexecution service that uses multiple computing systems on multiplephysical networks (e.g., multiple physical computing systems andnetworks within a data center). In at least some such embodiments,virtual computer networks to which computing nodes belong may beselected based on associated users, such as based on the computing nodesexecuting programs on behalf of a user or other entity.

As previously noted, a virtual computer network may in some embodimentsbe provided as an overlay network that uses one or more intermediatephysical networks as a substrate network, and one or more such overlayvirtual computer networks may be implemented over the substrate networkin various ways in various embodiments. For example, in at least someembodiments, communications between nodes of an overlay virtual computernetwork are managed by sending those communications over the substratenetwork without encapsulating the communications, such as by embeddingvirtual network address information for a computing node of the virtualcomputer network (e.g., the destination computing node's virtual networkaddress) in a larger physical network address space used for anetworking protocol of the one or more intermediate physical networks.As one illustrative example, a virtual computer network may beimplemented using 32-bit IPv4 network addresses, and those 32-bitvirtual network addresses may be embedded as part of 128-bit IPv6network addresses used by the one or more intermediate physicalnetworks, such as by re-headering communication packets or other datatransmissions (e.g., using Stateless IP/ICMP Translation, or SIIT), orotherwise modifying such data transmissions to translate them from afirst networking protocol for which they are configured to a distinctsecond networking protocol. As another illustrative example, both thevirtual computer network and substrate computer network may beimplemented using the same network addressing protocol (e.g., IPv4 orIPv6), and data transmissions sent via the provided overlay virtualcomputer network using virtual network addresses may be modified to usedifferent physical network addresses corresponding to the substratenetwork while the transmissions are sent over the substrate network, butwith the original virtual network addresses being stored in the modifieddata transmissions (e.g., in communication header fields) or otherwisetracked so that the data transmissions may be restored to their originalform when they exit the substrate network. In other embodiments, atleast some of the overlay computer networks may be implemented usingencapsulation of communications. Additional details related to SIIT areavailable at “Request For Comments 2765—Stateless IP/ICMP TranslationAlgorithm”, February 2000, at tools<dot>ietf<dot>org<slash>html<slash>rfc2765 (where <dot> and <slash> are replaced by thecorresponding characters with those names), which is hereby incorporatedby reference in its entirety. More generally, in some embodiments whenimplementing a first overlay network using a second substrate network,an N-bit network address that is specified for the first overlay networkin accordance with a first network addressing protocol may be embeddedas part of another M-bit network address that is specified for thesecond substrate network in accordance with a second network addressingprotocol, with “N” and “M” being any integers that correspond to networkaddressing protocols. In addition, in at least some embodiments, anN-bit network address may be embedded in another network address usingmore or less than N bits of the other network address, such as if agroup of N-bit network addresses of interest may be represented using asmaller number of bits (e.g., with L-bit labels or identifiers beingmapped to particular N-bit network addresses and embedded in the othernetwork addresses, where “L” is less than “N”).

Various benefits may be obtained from embedding virtual network addressinformation in substrate network addresses for an underlying substratenetwork, including enabling an overlay of the virtual computer networkon the substrate network without encapsulating communications orconfiguring physical networking devices of the substrate network, asdiscussed in greater detail below. Furthermore, other information maysimilarly be embedded in the larger physical network address space for acommunication between computing nodes in at least some embodiments andsituations, such as an identifier specific to a particular virtualcomputer network that includes those computing nodes (e.g., a virtualcomputer network for a user or other entity on whose behalf thosecomputing nodes operate), an identifier corresponding to a particularvirtual local area network, etc. Additional details related to provisionof such virtual computer networks via use of overlay networks areincluded below.

Furthermore, in addition to managing configured network topologies forprovided virtual computer networks, the CNS system may use the describedtechniques to provide various other benefits in various situations, suchas limiting communications to and/or from computing nodes of aparticular virtual computer network to other computing nodes that belongto that virtual computer network. In this manner, computing nodes thatbelong to multiple virtual computer networks may share parts of one ormore intermediate physical networks, while still maintaining networkisolation for computing nodes of a particular virtual computer network.In addition, the use of the described techniques also allows computingnodes to easily be added to and/or removed from a virtual computernetwork, such as to allow a user to dynamically modify the size of avirtual computer network (e.g., to dynamically modify the quantity ofcomputing nodes to reflect an amount of current need for more or lesscomputing resources). Furthermore, the use of the described techniquesalso supports changes to an underlying substrate network—for example, ifthe underlying substrate network is expanded to include additionalcomputing nodes at additional geographical locations, existing or newvirtual computer networks being provided may seamlessly use thoseadditional computing nodes, since the underlying substrate network willroute communications to and from the substrate network addresses forthose additional computing nodes in the same manner as for otherpreviously existing substrate network computing nodes. In at least someembodiments, the underlying substrate network may be of any size (e.g.,spanning multiple countries or continents), without regard to networklatency between computing nodes at different locations.

At least some such benefits may similarly apply for logical sub-networks(or “subnets”) that are specified for such a particular provided virtualcomputer network, with the substrate network functionality used toemulate various functionality corresponding to the specified logicalsubnets. For example, the use of the underlying substrate network mayenable different computing nodes assigned to a particular logical subnetto be located at any position within the substrate network, with thesubstrate network forwarding communications to destination computingnodes based on those destination computing nodes' substrate networkaddresses. As such, the substrate network may support any specifiedlogical subnets, without any configuration regarding such specifiedlogical subnets and without any other use of information about suchspecified logical subnets, and with the CNS system modules (e.g.,communication manager modules) instead managing the correspondingfunctionality from the logical edges of the substrate network where theCNS system modules connect to the substrate network. In addition,modules of the CNS system may similarly operate to limit communicationswithin a particular provided virtual computer network to occur onlybetween computing nodes assigned to particular logical subnets specifiedfor the provided computer network, if so configured, such as byauthorizing whether to forward particular communications to indicateddestination computing nodes and by directing communications toparticular computing nodes selected to act as destination computingnodes, so as to provide network isolation for computing nodes assignedto those specified logical subnets.

For illustrative purposes, some embodiments are described below in whichspecific types of computing nodes, networks, communications, networktopologies, and configuration operations are performed. These examplesare provided for illustrative purposes and are simplified for the sakeof brevity, and the inventive techniques may be used in a wide varietyof other situations, some of which are discussed below.

FIG. 1B is a network diagram illustrating an example embodiment ofconfiguring and managing communications between computing nodesbelonging to a virtual computer network, by overlaying the virtualcomputer network and the communications on one or more intermediatephysical networks in a manner transparent to the computing nodes of thevirtual computer network. In this example, the configuring and managingof the communications is facilitated by a system manager module, anetwork routing manager module, and multiple communication managermodules of an example embodiment of the CNS system. The example CNSsystem may be used, for example, in conjunction with a publiclyaccessible program execution service (not shown), or instead may be usedin other situations, such as with any use of virtual computer networkson behalf of one or more entities (e.g., to support multiple virtualcomputer networks for different parts of a business or otherorganization on a private network of the organization).

The illustrated example includes an example data center 190 withmultiple physical computing systems operated on behalf of the CNSsystem. The example data center 190 is connected to an internet 185external to the data center 190, which provides access to one or morecomputing systems 145 a via private network 165, to one or more otherglobally accessible data centers 160 that each have multiple computingsystems (not shown), and to one or more other computing systems 145 b.The internet 185 may be, for example, a publicly accessible network ofnetworks (possibly operated by various distinct parties), such as theInternet, and the private network 165 may be, for example, a corporatenetwork that is wholly or partially inaccessible from computing systemsexternal to the private network 165. Computing systems 145 b may be, forexample, home computing systems or mobile computing devices that eachconnects directly to the Internet (e.g., via a telephone line, cablemodem, a Digital Subscriber Line (“DSL”), cellular network or otherwireless connection, etc.).

The example data center 190 includes a number of example physicalcomputing systems 106 a-106 d and 155 a-155 n, as well as aCommunication Manager module 150 that executes on one or more othercomputing systems or devices (not shown) to manage communications forthe associated computing systems 155 a-155 n, an external CommunicationManager module 108 that executes on one or more other computing systemsor devices (not shown) to manage external communications that pass inand out of the data center 190 over the internet 185, a System Managermodule 110 that executes on one or more computing systems (not shown),and a Network Routing Manager module 170 that executes on one or morecomputing systems (not shown). While not illustrated here, in someembodiments one or more other modules may also be present and executingon one or more computing systems or devices of the data center 190, andthe System Manager module 110 and/or the Network Routing Manager module170 may have one or more associated Communication Manager modules thatmanage information sent to and from the modules 110 and 170. In thisexample, each physical computing system 106 a-106 d hosts multiplevirtual machine computing nodes and includes an associated virtualmachine (“VM”) communication manager module (e.g., as part of a virtualmachine hypervisor monitor for the physical computing system), such asVM Communication Manager module 109 a and multiple virtual machines 107a on host computing system 106 a, and such as VM Communication Managermodule 109 d and multiple virtual machines 107 d on host computingsystem 106 d. Physical computing systems 155 a-155 n do not execute anyvirtual machines in this example, and thus may each act as a computingnode that directly executes one or more software programs on behalf of auser. The Communication Manager module 150 that manages communicationsfor the associated computing systems 155 a-155 n may be implemented aspart of various types of devices, such as, for example, a proxycomputing device, a firewall device, or a networking device (e.g., aswitch, router, hub, etc.) through which communications to and from thephysical computing systems travel. Similarly, the Communication Managermodule 108 that manages communications between the data center 190 andthe internet 185 may be implemented as part of various types of devices,such as, for example, a firewall device, a network address translation(“NAT”) device, one or more load balancer devices, the edge router 127c, or another networking device. In other embodiments, all or none ofthe physical computing systems at the data center may host virtualmachines.

This example data center 190 further includes multiple physicallyprovided networking devices, such as switches 119 a-119 b, edge routerdevices 127 a-127 c, and core router devices 132 a-132 c. Switch 119 ais part of a physical sub-network that includes physical computingsystems 106 a-106 c, and is connected to edge router 127 a. Switch 119 bis part of a distinct physical sub-network that includes physicalcomputing systems 106 d and 155 a-155 n, as well as the computingsystems providing the Communication Manager module 150, the SystemManager module 110, and the Network Routing Manager module 170, and isconnected to edge router 127 b. The physical sub-networks established byswitches 119 a-119 b, in turn, are connected to each other and othernetworks (e.g., the internet 185) via an intermediate interconnectionnetwork 122, which includes the edge routers 127 a-127 c and the corerouters 132 a-132 c. The edge routers 127 a-127 c provide gatewaysbetween two or more physical sub-networks or networks. For example, edgerouter 127 a provides a gateway between the physical sub-networkestablished by switch 119 a and the interconnection network 122, whileedge router 127 c provides a gateway between the interconnection network122 and internet 185 (e.g., via the module 108). The core routers 132a-132 c manage communications within the interconnection network 122,such as by routing or otherwise forwarding packets or other datatransmissions as appropriate based on characteristics of such datatransmissions (e.g., header information including source and/ordestination addresses, protocol identifiers, etc.) and/or thecharacteristics of the interconnection network 122 itself (e.g., routesbased on the physical network topology, etc.).

The illustrated System Manager module and Communication Manager modulesmay perform at least some of the described techniques in order toconfigure, authorize and otherwise manage communications sent to andfrom associated computing nodes, including to support providing variousvirtual networking functionality for one or more virtual computernetworks that are provided using various of the computing nodes, and/orto support providing various emulated functionality for one or morevirtual networking devices that are configured for one or more suchprovided virtual computer networks. Such functionality may further besupported in at least some embodiments by one or more Network RoutingManager modules, such as Network Routing Manager module 170, whichassist in determining and configuring routing information to be used tosupport provided virtual computer networks, as discussed in greaterdetail below. In this example, Communication Manager module 109 amanages associated virtual machine computing nodes 107 a, CommunicationManager module 109 d manages associated virtual machine computing nodes107 d, and each of the other Communication Manager modules 109 and 150may similarly manage communications for a group of one or more otherassociated computing nodes. The illustrated Communication Managermodules may configure communications between computing nodes so as tooverlay one or more particular virtual networks over one or moreintermediate physical networks that are used as a substrate network,such as over the interconnection network 122, and so as to support oneor more virtual networking devices that are not physically provided forsuch an overlaid particular virtual network. Furthermore, a particularvirtual computer network may optionally be extended beyond the datacenter 190 in some embodiments, such as if one or more other datacenters 160 also provide computing nodes that are available for use bythe example CNS system, and the particular virtual computer networkincludes computing nodes at two or more such data centers at two or moredistinct geographical locations. Multiple such data centers or othergeographical locations of one or more computing nodes may beinter-connected in various manners, including the following: directlyvia one or more public networks in a non-private manner, or via one ormore private connections, not shown (e.g., a dedicated physicalconnection that is not shared with any third parties, such as a leasedline; a VPN or other mechanism that provides the private connection overa public network; etc.). In addition, while not illustrated here, othersuch data centers or other geographical locations may each include oneor more other Communication Manager modules that manage communicationsfor computing systems at that data center or other geographicallocation, such as Communication Manager modules similar to module 108that manage external communications for their data centers.

In addition, a particular virtual computer network may optionally beextended beyond the data center 190 in other manners in otherembodiments, such as based on one or more other Communication Managermodules external to the data center 190 (e.g., if another CommunicationManager module is made part of private network 165, so as to managecommunications for computing systems 145 a over the internet 185 andprivate network 165; etc.). Thus, for example, if an organizationoperating private network 165 desires to virtually extend its privatecomputer network 165 to one or more of the computing nodes of the datacenter 190, it may do so by implementing one or more CommunicationManager modules as part of the private network 165 (e.g., as part of theinterface between the private network 165 and the internet 185)—in thismanner, computing systems 145 a within the private network 165 maycommunicate with those data center computing nodes as if those datacenter computing nodes were part of the private network. In otherembodiments, the private computer network 165 may instead be extended toone or more computing nodes of the data center 190 by the module 108 ofthe data center 190 managing the communications between computing nodesof the private network 165 and particular data center 190 computingnodes.

Thus, as one illustrative example, one of the virtual machine computingnodes 107 a on computing system 106 a (in this example, virtual machinecomputing node 107 a 1) may be part of the same provided virtualcomputer network as one of the virtual machine computing nodes 107 d oncomputing system 106 d (in this example, virtual machine computing node107 d 1), and may further both be assigned to a specified logical subnetof that virtual computer network that includes a subset of the computingnodes for that virtual computer network, such as with the IPv4networking protocol being used to represent the virtual networkaddresses for the virtual computer network. The virtual machine 107 a 1may then direct an outgoing communication (not shown) to the destinationvirtual machine computing node 107 d 1, such as by specifying a virtualnetwork address for that destination virtual machine computing node(e.g., a virtual network address that is unique for the local broadcastdomain of the specified subnet). The Communication Manager module 109 areceives the outgoing communication, and in at least some embodimentsdetermines whether to authorize the sending of the outgoingcommunication, such as based on previously obtained information aboutthe sending virtual machine computing node 107 a 1 and/or about thedestination virtual machine computing node 107 d 1 (e.g., informationabout virtual computer networks and/or entities with which the computingnodes are associated, information about any specified logical subnets towhich the computing nodes belong, etc.), and/or by dynamicallyinteracting with the System Manager module 110 (e.g., to obtain anauthorization determination, to obtain some or all such information,etc.). By not delivering unauthorized communications to computing nodes,network isolation and security of entities' virtual computer networks isenhanced.

If the Communication Manager module 109 a determines that the outgoingcommunication is authorized (or does not perform such an authorizationdetermination), the module 109 a determines the actual physical networklocation corresponding to the destination virtual network address forthe communication. For example, the Communication Manager module 109 amay determine the actual destination network address to use for thevirtual network address of the destination virtual machine 107 d 1 bydynamically interacting with the System Manager module 110, or may havepreviously determined and stored that information (e.g., in response toa request from the sending virtual machine 107 a for information aboutthat destination virtual network address, such as a request that thevirtual machine 107 a 1 specifies using Address Resolution Protocol, orARP). The Communication Manager module 109 a then re-headers orotherwise modifies the outgoing communication so that it is directed toCommunication Manager module 109 d using an actual substrate networkaddress, such as if Communication Manager module 109 d is associatedwith a range of multiple such actual substrate network addresses. FIGS.2A-2E provide examples of doing such communication management in someembodiments, including to emulate functionality corresponding to one ormore virtual networking devices specified for the virtual network thatare not physically provided.

When Communication Manager module 109 d receives the communication viathe interconnection network 122 in this example, it obtains the virtualdestination network address for the communication (e.g., by extractingthe virtual destination network address from the communication), anddetermines to which of the virtual machine computing nodes 107 d managedby the Communication Manager module 109 d that the communication isdirected. The Communication Manager module 109 d next determines whetherthe communication is authorized for the destination virtual machinecomputing node 107 d 1, with examples of such authorization activitiesdiscussed in further detail in the examples of FIGS. 2A-2E andelsewhere. If the communication is determined to be authorized (or theCommunication Manager module 109 d does not perform such anauthorization determination), the Communication Manager module 109 dthen re-headers or otherwise modifies the incoming communication so thatit is directed to the destination virtual machine computing node 107 d 1using an appropriate virtual network address for the virtual computernetwork, such as by using the sending virtual machine computing node 107a 1's virtual network address as the source network address and by usingthe destination virtual machine computing node 107 d 1's virtual networkaddress as the destination network address. The Communication Managermodule 109 d then forwards the modified communication to the destinationvirtual machine computing node 107 d 1. In at least some embodiments,before forwarding the incoming communication to the destination virtualmachine, the Communication Manager module 109 d may also performadditional steps related to security, as discussed in greater detailelsewhere.

In addition, while not illustrated in FIG. 1B, in some embodiments thevarious Communication Manager modules may take further actions toprovide virtual networking functionality corresponding to a specifiednetwork topology for the provided virtual computer network (e.g., forone or more virtual networking devices for the provided virtual computernetwork), such as by managing communications between computing nodes ofthe provided virtual computer network in specified manners and byresponding to other types of requests sent by computing nodes of thevirtual computer network. For example, although being separated fromcomputing node 107 a 1 on physical computing system 106 a by theinterconnection network 122 in the example embodiment of FIG. 1B,virtual machine computing node 107 d 1 on physical computing system 106d may be configured to be part of the same logical sub-network of thevirtual computer network as computing node 107 a 1 (e.g., to not beseparated by any specified router devices). Conversely, despite thephysical proximity of virtual machine computing node 107 c 1 on physicalcomputing system 106 c to virtual machine computing node 107 a 1 onphysical computing system 106 a (i.e., being part of the same physicalsub-network without any intervening physical router devices) in theexample embodiment of FIG. 1B, computing node 107 c 1 may be configuredto be part of a distinct logical sub-network of the virtual computernetwork from that of computing node 107 a 1 (e.g., may be configured tobe separated by one or more specified router devices, not shown, whichin this example are virtual router devices that are not physicallyprovided for the virtual computer network). If computing nodes 107 a 1and 107 d 1 are configured to be part of the same logical sub-network,the previous example of sending a communication from computing node 107a 1 to computing node 107 d 1 may be performed in the manner previouslydescribed, without emulating the actions of any intervening virtualrouter devices (despite the use of multiple physical router devices inthe substrate interconnection network 122 for forwarding thecommunication), since computing nodes 107 a 1 and 107 d 1 are configuredto be part of single sub-network in the specified network topology.

However, if computing node 107 a 1 sends an additional communication tocomputing node 107 c 1, the Communication Manager modules 109 a and/or109 c on the host computing systems 106 a and 106 c may performadditional actions that correspond to one or more virtual router devicesconfigured in the specified network topology to separate the computingnodes 107 a 1 and 107 c 1. For example, the source computing node 107 a1 may send the additional communication in such a manner as to initiallydirect it to a first of the virtual router devices that is configured tobe local to computing node 107 a 1 (e.g., by including a virtualhardware address in the header of the additional communication thatcorresponds to that first virtual specified router device), with thatfirst virtual router device being expected to forward the additionalcommunication on toward the destination computing node 107 c 1 via thespecified logical network topology. If so, the source CommunicationManager module 109 a may detect the forwarding of the additionalcommunication to the virtual first router device (e.g., based on thevirtual hardware address used in the header of the additionalcommunication), or otherwise be aware of the configured network topologyfor the virtual computer network, and may take actions to emulatefunctionality of some or all of the virtual specified router devicesthat are configured in the specified network topology to separate thecomputing nodes 107 a 1 and 107 c 1. For example, each virtual routerdevice that forwards the additional communication may be expected totake actions such as modifying a TTL (“time to live”) hop value for thecommunication, modify a virtual destination hardware address that isspecified for the communication to indicate the next intendeddestination of the additional communication on a route to thedestination computing node, and/or otherwise modify the communicationheader. If so, the source Communication Manager module 109 a may performsome or all of those actions before forwarding the additionalcommunication directly to the destination Communication Manager module109 c over the substrate network (in this case, via physical switchdevice 119 a) for provision to destination computing node 107 c 1.Alternatively, some or all such additional actions to provide thevirtual networking functionality for the sent additional communicationmay instead be performed by the destination Communication Manager module109 c after the additional communication is forwarded to theCommunication Manager module 109 c by the Communication Manager module109 a.

By providing virtual networking functionality using the describedtechniques, the CNS system provides various benefits. For example,because the various Communication Manager modules manage the overlayvirtual network and may emulate functionality of virtual networkingdevices, specified networking devices and other network topology do notneed to be physically implemented for virtual computer networks beingprovided, and thus corresponding modifications are not needed to theinterconnection network 122 or switches 119 a-119 b to supportparticular configured network topologies. Nonetheless, if the computingnodes and software programs of a virtual computer network have beenconfigured to expect a particular network topology for the providedvirtual computer network, the appearance and functionality of thatnetwork topology may nonetheless be transparently provided for thosecomputing nodes by the described techniques.

In addition, as previously noted, the described techniques includeperforming various additional actions in at least some embodiments tosupport networking devices specified for managed computer networks, suchas for virtual networking devices that are not physically provided andwhose functionality is emulated, with the modules of a configurablenetwork service being configured to perform various automated operationsto support such emulated functionality.

Thus, in at least some embodiments in which a network-accessibleconfigurable network service supports virtual networking devices thatare not physically provided for managed computer networks, theconfigurable network service may perform various actions to enable thevirtual networking devices to participate in routing protocols in thesame manner or a similar manner as would an actual physically providednetworking device of the managed computer network. However, in at leastsome situations in which an embodiment of a configurable network serviceprovides a managed computer network as a virtual computer networkoverlaid on a substrate network, at least some such routing informationfor the virtual computer network is not needed by the configurablenetwork service to determine how to send network communications to theirfinal destinations. For example, if the configurable network servicetracks the location in the substrate network of each computing node thatis part of the virtual computer network, then network communications mayin some situations be forwarded to a destination computing node over thesubstrate network without using routing information for the virtualcomputer network (although optionally modifying the networkcommunications to make changes that would have been made by appropriatevirtual networking devices of the virtual computer network if they werephysically provided and used to forward the network communications, asdiscussed in greater detail elsewhere). However, the network topologyand other configuration for the virtual computer network may influencehow network communications are forwarded between computing nodes in atleast some other situations. As a simple illustrative example, considera virtual computer network that is configured into two logical subnets,with one of the virtual network computing nodes acting as a firewallthat has distinct network interfaces in each of the two logical subnetsto facilitate handling inter-subnet network communications by performinganalysis of and possibly interdiction of such network communicationsthat pass through it. In such a situation, if the sending anddestination computing nodes for a network communication are in separatelogical subnets, the configurable network service does not merelyforward the communication over the substrate network directly to thedestination computing node—instead, the logical network topology for thevirtual computer network dictates that the configurable network serviceforward the communication from the sending computing node over thesubstrate network to the intermediate firewall device computing node forhandling, with the communication sent to the network interface of thefirewall device computing node that is connected to the sendingcomputing node's logical subnet. If the firewall device computing nodepasses the communication on (optionally with modifications), theconfigurable network service then further forwards the communicationover the substrate network from the other network interface of thefirewall device computing node in the other logical subnet to theultimate destination computing node.

In at least some embodiments and situations, the routing informationthat is forwarded over a managed virtual computer network by computingnodes of the virtual computer network may be of use to the configurablenetwork service in various manners, including to update network topologyinformation for the virtual computer network. For example, as acontinuation of the prior example having a virtual computer network withtwo logical subnets, the client associated with the virtual computernetwork may have initially configured the network topology of thevirtual computer network to have the two logical subnets and to havethem separated by at least one specified virtual network router or othervirtual networking device, but not have initially configured anyintermediate firewall device between the logical subnets. As discussedin greater detail elsewhere, such initial configuration information maybe supplied by the client using an API of the configurable networkservice, or instead in other manners. In addition, as noted above, insuch a situation, communications that are initially forwarded betweencomputing nodes in separate logical subnets may be directly sent to thedestination computing node over the substrate network, without havingthe communications pass through any intermediate computing nodes of thevirtual computer network. However, after the virtual computer network isin use and one or more such network communications have been forwardedbetween computing nodes, the client may dynamically make changes to thevirtual computer network that affect the network topology of the virtualcomputer network. As one example, the client may executefirewall-related software on one of the computing nodes of the virtualcomputer network, and that new firewall computing node may begin to actas the firewall device in the prior example, so as to analyzecommunications that pass between computing nodes of the two logicalsubnets. In order to detect this network topology change and takeappropriate actions, the configurable network service may interceptrouting-related communications that are sent by the new firewall devicein accordance with whatever routing protocol is used for the virtualcomputer network (e.g., announcement routing communications that aresent by the firewall computing node over the network interface for afirst of the logical subnets that it can manage communications for thecomputing nodes in the other logical subnet, and similar announcementrouting communications sent over the other network interface for thecomputing nodes in the first logical subnet), and use the routinginformation included in those routing-related communications to detectthe location and operation of the new firewall device, such as based onthe network interfaces of the new firewall device in the two logicalsubnets. In this manner, the configurable network service will determineto route subsequent network communications between the logical subnetsvia the firewall device, using the network interfaces of the firewalldevice, in the manner previously discussed.

Various routing protocols have been defined and are in use, and in atleast some embodiments, the configurable network service may support theuse of some or all of the various routing protocols (e.g., with a firstmanaged virtual computer network that is provided by the configurablenetwork service using a first routing protocol, with a distinct secondmanaged virtual computer network that is provided by the configurablenetwork service using one or more distinct second routing protocols,etc., and with the configurable network service extracting and usingrouting information sent using the various routing protocols for thevarious managed virtual computer networks). A non-exclusive list ofpredefined routing protocols that may be supported by embodiments of theconfigurable network service includes the following: OSPF (Open ShortestPath First); IS-IS (Intermediate System to Intermediate System); BGP(Border Gateway Protocol); RIP (Routing Information Protocol); IGRP(Interior Gateway Routing Protocol); EIGRP (Enhanced Interior GatewayRouting Protocol); RSVP (Resource ReSerVation Protocol); GARP(gratuitous address resolution protocol); etc. Moreover, by supportingmultiple routing protocols that are in use by a single managed virtualcomputer network, the configurable network service may in someembodiments enable more consistent and/or accurate routing informationto be provided to computing nodes of the managed virtual computernetwork that participate in one or more of those routing protocols. Forexample, consider a situation in which the managed virtual computernetwork has a first computing node that uses a first routing protocol(e.g., BGP), which is connected to a second computing node that uses adistinct second routing protocol (e.g., OSPF), and the second computingnode is connected to one or more other third computing nodes that alsouse the first routing protocol. In this situation, routing informationthat the first computing node receives using the first routing protocolwould typically be converted to the second routing protocol for thesecond computing node, and then later converted back to the firstrouting protocol for the third computing node(s)—however, suchconversions may result in the routing information that is provided tothe third computing nodes using the first routing protocol beingdifferent from the routing information that was originally provided tothe first computing node using the first routing protocol (e.g., withrespect to cost information for particular routing paths or otherinformation related to connectivity). Instead, some embodiments of theconfigurable network service may instead manage the routing informationthat is provided to the third computing nodes using the first routingprotocol to ensure that it is the same as or otherwise consistent withthe routing information received by the first computing node. Moregenerally, if a particular group of routing information is sent tovarious computing nodes of a managed virtual computer network using twoor more routing protocols, some embodiments of the configurable networkservice may manage that routing information to assist in maintainingconsistency between the various routing protocols' routing information.

In addition, various types of computing nodes may send routinginformation to the virtual networking devices of managed virtualcomputer networks in various embodiments. Such computing nodes may beprovided by the configurable network service and be part of the virtualcomputer networks for which they send routing information (e.g., undercontrol of the client for whom the virtual computer network is beingprovided), and/or may include other types of computing nodes/devicesthat are not part of the virtual computer networks. For example, if aclient operates a separate computer network that interacts with amanaged virtual computer network provided by an embodiment of theconfigurable network service (e.g., a remote computer network for whichthe managed virtual computer network is an extension), such as mayinteract via one or more VPN connections or other connections, one ormore computing nodes/devices of the other computer network may in someembodiments and situations send routing-related communications thatinclude routing information of potential use to the configurable networkservice. In addition, the types of computing nodes/devices that sendsuch routing-related communications may be actual network routers orother actual networking devices (e.g., provided by and under control ofa client for whom a managed virtual computer network is being provided),and/or various other types of computing nodes that may be configured(e.g., via software) to act as network appliances and provide at leastsome types of functionality related to handling network communications(e.g., firewalls; proxy devices; load balancers; VPN endpoints;intrusion detection and/or prevention systems; etc.), such as computingnodes of a managed virtual computer network provided for a client thatare configured by the client to operate as network appliances for thevirtual computer network.

In addition, in order to facilitate handling of routing-relatedcommunications from other computing nodes to virtual networking devicesof managed virtual computer networks, embodiments of the configurablenetwork service may further take various actions to represent thevirtual networking devices to the other computing nodes. In particular,the configurable network service may in some embodiments assign at leastone network address for a virtual computer network to each virtualnetworking device for the virtual computer network, so as to enablerouting-related communications to be directed to the virtual networkingdevices using those assigned virtual network addresses. For example, ifa client specifies configuration information that indicates multiplevirtual network addresses to be used with a managed virtual computernetwork, each of the multiple computing nodes of the managed virtualcomputer network may be assigned at least one of those multiple virtualnetwork addresses, and each of the virtual networking devices may beassigned at least one other of those virtual network addresses that isdistinct from the virtual network addresses used for the multiplecomputing nodes (and for the other virtual networking devices). Asdiscussed in greater detail elsewhere, when network communications arereceived by the configurable network service that are directed to avirtual network address assigned to a virtual networking device, theconfigurable network service may intercept the communications, extractthe routing information from the communications, and use the extractedrouting information in various manners. In addition, in at least someembodiments, the configurable network service may further take actionsto send routing information to other computing nodes of a managedvirtual computer network (e.g., to send routing-related communicationsthat are spoofed to be or otherwise identified as being from virtualnetworking devices of the managed virtual computer network), such as inaccordance with the routing protocol(s) used for that managed virtualcomputer network. Such routing information may be sent to computingnodes of the managed virtual computer network, for example, for one ormore of the following reasons: to forward routing information directedto the virtual networking devices from other computing nodes; to provideupdates about routing-related changes that a client has made to themanaged virtual computer network by modifying configuration informationfor the virtual computer network, such as modifying network topologyinformation via an API of the configurable network service; etc. Thus,in at least some embodiments and situations, clients to whom managedvirtual computer networks are provided may be unaware that the virtualnetworking devices of their managed virtual computer networks are notphysically provided devices with which the other computing nodes areactually interacting, based on the actions of the configurable networkservice to emulate various functionality of those virtual networkingdevices, including corresponding to routing-related communications.

As previously noted, routing-related communications that are directed tovirtual networking devices of managed virtual computer networks may beintercepted and used by the configurable network service in varioussituations. For example, in at least some embodiments, each computingnode of a managed virtual computer network has an associatedcommunication manager module of the configurable network service thatmanages communications for the computing node. In such embodiments,those communication manager modules may perform at least some of thedescribed techniques. For example, when such a communication managermodule receives a routing-related communication from one of thecomputing nodes that it manages that is directed to a virtual networkingdevice, the communication manager module may identify that intendeddestination as being a virtual networking device (e.g., based on thedestination virtual network address used, based on a particular routingprotocol used for the communication, etc.), and instead process therouting-related communication so as to enable the routing information inthe communication to be used by the configurable network service. Insome embodiments, the communication manager module may extract therouting information from the routing-related communication, and forwardthe extracted routing information to another module of the configurablenetwork service for further processing (e.g., to a Network RoutingManager, or NRM, module). Furthermore, if a response message wouldtypically be provided by the intended destination virtual networkingdevice based on the routing protocol in use, the local communicationmanager module may generate and provide a spoofed response to the senderas appropriate that is indicated to be from the virtual networkingdevice. In other embodiments, the various communication manager modulesmay instead forward the routing-related communications to the NRM module(or other module) for similar processing by the NRM module, such as toenable the NRM module to extract the routing information. As previouslynoted, in at least some embodiments, the communication manager modulesand/or the NRM module may support multiple distinct routing protocols.

When the NRM module receives routing information from a routing-relatedcommunication directed to a virtual networking device of a managedvirtual computer network (whether directly from a communication managermodule, or after extracting the routing information from arouting-related communication received from the communication managermodule), the NRM module analyzes the routing information to determine ifit indicates any changes with respect to the previously known networktopology and connectivity information for the managed virtual computernetwork, such as that would affect how network communications are routedfor the managed virtual computer network (e.g., whether there areintermediate computing nodes of the managed virtual computer networkthrough which particular network communications are to be routed). Ifnetwork topology changes are identified for the managed virtual computernetwork, the NRM module takes actions to correspondingly update therouting information that is used for the managed virtual computernetwork. In particular, in at least some embodiments, the communicationmanager modules associated with computing nodes of managed virtualcomputer networks perform various actions to manage outgoing networkcommunications from their associated computing nodes, such as byforwarding those outgoing network communications to appropriatedestinations—in such embodiments, the communication manager module(s)that are associated with computing nodes of the managed virtual computernetwork that are affected by the identified network topology change maybe updated to handle network communications for those associatedcomputing nodes in a manner consistent with the identified networktopology change. For example, in the previously discussed situation inwhich a firewall device computing node of a managed virtual computernetwork is dynamically added between two logical subnets of that managedvirtual computer network, the communication manager module(s) associatedwith the computing nodes of those two logical subnets are updated tohandle new network communications between the two logical subnets in acorresponding manner. Thus, for a particular computing node in a firstof the two logical subnets, the associated communication manager moduleis updated so that destination virtual network addresses for othercomputing nodes of the first logical subnet will continue to be mappedto the substrate network locations of those other computing nodes (sothat communications directed to those computing nodes will be forwardeddirectly to those computing nodes over the substrate network), whiledestination virtual network addresses for computing nodes of the othersecond logical subnet will newly be mapped to the substrate networklocation of the firewall device computing node for further handling (sothat communications directed to those computing nodes will be forwardedover the substrate network to the firewall device for intermediatehandling).

Thus, the NRM module may obtain routing information from routing-relatedcommunications directed to virtual networking devices of a managedvirtual computer network, and use the routing information to update aconfigured network topology of the managed virtual computer network. Inaddition, in at least some embodiments, the NRM module may take furtheractions to facilitate the routing of network communications for such amanaged virtual computer network over an underlying substrate network inan appropriate manner. For example, when the managed virtual computernetwork is first created, the NRM module may use any configured networktopology information for the managed virtual computer network (e.g., asspecified via an API of the configurable network service) in a similarmanner to determine whether and when any network communications betweencomputing nodes of the managed virtual computer network are to bedirected through one or more intermediate computing nodes of the managedvirtual computer network based on the configured network topologyinformation, and configure the communication manager modules for thecomputing nodes of the managed virtual computer network in acorresponding manner. Similarly, if configured network topologyinformation for the managed virtual computer network is updated afterthe managed virtual computer network is in use, such as viaconfiguration information explicitly supplied by the associated clientto the system manager module of the configurable network service (e.g.,via an API and/or GUI provided by the configurable network service), theNRM module may similarly identify any network topology changes thataffect the routing of network communications, and update the appropriatecommunication manager modules to handle those network topology changesin an appropriate manner.

Furthermore, in addition to managing the routing of networkcommunications via the substrate network, the NRM module may use theobtained routing information for a managed virtual computer network inother manners in at least some embodiments. For example, when the NRMmodule identifies a change in network topology for the managed virtualcomputer network (whether from obtained routing information or from newnetwork topology configuration information received from the client),the NRM module may in some embodiments generate routing-relatedcommunications that include corresponding routing information in orderto update other computing nodes of the managed virtual computer network.Thus, for each of one or more of the computing nodes of the managedvirtual computer network that participate in the defined routingprotocol for the managed virtual computer network and that performfunctionality that involves handling network communications, the NRMmodule may instruct a communication manager module that manages thatcomputing node to supply that computing node with the routinginformation corresponding to the network topology change, such as bysending a routing-related communication to that communication managermodule that is destined for that computing node. Such a communicationsupplied to such a computing node of the managed virtual computernetwork may, for example, have spoofed sender information so that thecommunication appears to come from a virtual networking device of themanaged virtual computer network, or may instead be sent in othermanners in other embodiments.

When providing routing information for a managed virtual computernetwork to computing nodes of the managed virtual computer network, aswell as when providing substrate network routing information toparticular communication manager modules that manage communications forcomputing nodes of the managed virtual computer network, the NRM modulemay determine the routing information in a manner that is contextualizedto the recipient, as different computing nodes and/or communicationmanager modules typically route network communications differentlydepending on their network location (e.g., their relative location inthe configured network topology of the managed virtual computernetwork). Nonetheless, by having a centralized location for the variousinformation about the network topology of a managed virtual computernetwork, including in some embodiments to have a combination ofclient-supplied configuration information and dynamically suppliedrouting information from the actual managed virtual computer network,the NRM module may in some embodiments use the aggregate information todetermine consistent routing information to be used by the variouscomputing nodes of the managed virtual computer network (e.g., todetermine network routing paths for particular computing nodes to usethat are globally optimized for the managed virtual computer network,including in light of network traffic from routing paths used by othercomputing nodes of the managed virtual computer network that facilitatehandling communications for the managed virtual computer network). Thus,rather than having each such network router and other computing node ofthe managed virtual computer network independently attempt to determinebest routing paths in a localized manner, the NRM module in suchembodiments may coordinate the routing information used across themanaged virtual computer network, such as to maximize or otherwiseenhance any desired criteria regarding characteristics of the operationof the managed virtual computer network (e.g., to balance networktraffic passing between multiple particular computing nodes, to minimizeor maximize network traffic passing through a particular computing noderelative to one or more other computing nodes, etc.).

Furthermore, in at least some embodiments, the configurable networkservice may perform various actions to accomplish other goals. Forexample, in order to limit undesirable behavior, the configurablenetwork service may in some embodiments limit the types of changes tonetwork topology information that some or all computing nodes of amanaged virtual computer network are allowed to make. As one example, insome embodiments, some types of network topology changes may be limitedto being made via configuration information supplied to the systemmanager module of the configurable network service by the client, ratherthan via routing information from computing nodes of the managed virtualcomputer network, or vice versa. Furthermore, the configurable networkservice may limit the rate and/or total quantity of routing-relatedcommunications that some or all computing nodes of a managed virtualcomputer network are allowed to make. Such limiting of routing-relatedcommunications and/or other filtering of network topology changes may beperformed in various manners, including by the communication managermodules associated with the computing nodes that attempt to send suchrouting-related communications or otherwise make such network topologychanges (e.g., by discarding routing-related communications ifappropriate), and/or by the NRM module (e.g., by not updating theconfigured network topology for a managed virtual computer network inaccordance with disallowed network topology changes and/orrouting-related communications, such as by not updating communicationmanager modules for the computing nodes of the managed virtual computernetwork to support such changes).

Thus, a configurable network service may take various actions to supporta configured logical network topology for a managed computer network,such as to include one or more virtual networking devices as part of amanaged virtual computer network. As one example, the CNS system may usemultiple communication manager modules to transparently managecommunications sent by and to the computing nodes of the virtualcomputer network in a manner that emulates functionality that would beprovided by one or more virtual networking devices if they werephysically implemented for the virtual computer network and wereconfigured to route or otherwise forward the communications inaccordance with the specified network topology. Furthermore, theconfigurable network service may use multiple communication managermodules to emulate responses to networking requests made by computingnodes in the manner of a local physical networking device, such as torespond to ping requests, SNMP (“Simple Network Management Protocol”)queries, etc. In this manner, the configurable network service mayprovide virtual networking functionality that corresponds to a specifiednetwork topology for a managed virtual computer network that is providedby the configurable network service, but without the computing nodes ofthe virtual computer network (or the associated client user or otherentity) being aware that the actual computer network is not configuredin a typical manner to support the network topology and to physicallyinclude the virtual networking devices. Furthermore, as described ingreater detail below, in at least some embodiments, multiple modules ofthe configurable network service may operate together in a distributedmanner to provide functionality corresponding to a particular virtualnetworking device, such that no single module or physical device issingly responsible for emulating a particular virtual networking device.

As previously noted, the providing of networking functionality by aconfigurable network service for a managed computer network provided toa client may include using routing information for the managed computernetwork in various manners. For example, the configurable networkservice may allow clients to configure target virtual network addressesof various types to be used with managed computer networks provided tothe clients, such as part of the initial configuration of a managedcomputer network and/or based on dynamic configuration that is performedwhile a managed computer network is in use. In particular, in someembodiments and situations, a client may configure that a particulartarget virtual network address will be used in a particular indicatedmanner within a managed computer network of the client, and/or thatparticular computing nodes may be associated with target virtual networkaddresses in various manners. A non-exclusive list of some types ofconfigurations that a client may perform for a managed computer networkinclude the following: that a particular target virtual network addressis an anycast network address for the managed computer network; thatparticular computing nodes of the managed computer network may or maynot be part of the group of computing nodes associated with an anycasttarget network address (e.g., a particular anycast target networkaddress, any anycast target network address, etc.), or more generallycriteria for use in determining whether to allow a computing node to bepart of such a group of associated computing nodes; that a particulartarget virtual network address is designated as being migratable withinthe managed computer network, such that different computing nodes mayserially claim ownership of or otherwise be associated with that targetvirtual network address; criteria for use in determining whether avirtual network address whose ownership is dynamically claimed by acomputing node but that has not been designated as a migratable targetnetwork address is allowed to be used as such a migratable targetnetwork address; that particular computing nodes of the managed computernetwork may or may not be allowed to claim ownership of a migratabletarget network address (e.g., a particular migratable target networkaddress, any migratable target network address, etc.), or more generallycriteria for use in determining whether to allow a computing node toclaim ownership of a migratable target network address; etc.

In addition, computing nodes of a managed computer network maydynamically signal particular types of uses of indicated target networkaddresses based on routing information in various manners in variousembodiments. For example, for a computing node that participates in oneor more routing protocols for a managed computer network (e.g., acomputing node that acts as an intermediate computing node to facilitatehandling of communications intended for other computing nodes), thecomputing node may send one or more routing communications in accordancewith those routing protocols. More generally, in at least someembodiments and situations, any computing node of the managed computernetwork may dynamically signal ownership of or other association with atarget virtual network address using one or more signaling mechanisms,such as to perform a GARP (gratuitous ARP) in which the computing nodeannounces the target virtual network address as being its own virtualnetwork address. In addition, a particular computing node maydynamically announce via a routing communication that it is newlyassociated with a particular target virtual network address, so thatcommunications directed to that target virtual network address will berouted to that computing node, but in some situations that computingnode may do so on behalf of one or more other computing node, such thatthe announcing computing node acts as an intermediate destination thatwill forward such communications directed to that target network addresson to the one or more other computing nodes that actually use the targetnetwork address. In other embodiments and situations, a computing nodethat dynamically announces a target virtual network address via arouting communication is the computing node that claims ownership of orother association with that target virtual network address.

As previously noted, one example of a type of use of target virtualnetwork addresses is for an anycast network address, with the anycastnetwork address having a group of one or more associated virtual networkaddress that serve as alternatives for communications directed to thatanycast network address. Furthermore, as noted elsewhere, computingnodes of a managed computer network may dynamically join or leave suchan anycast network address group in at least some embodiments, includingvia routing information included in routing communications such by thosecomputing nodes. For example, in some situations, the group of computingnodes associated with an anycast network group may be configured toinclude at least one computing node in each of multiple data centers orother geographical locations over which a managed computer networkspans, such that a nearby computing node from the group may be selected.In other situations, the group of computing nodes associated with ananycast network group may be configured to include multiple computingnodes for the purpose of load balancing and/or availability if aparticular one of the group computing nodes becomes unavailable. Whenmultiple computing nodes are associated with a group, a particularcomputing node to use from the group may be selected in various manners,such as by considering various factors (e.g., to select the computingnode that is the nearest, the most underutilized, etc.), in a dynamicmanner at the time of use for some or all uses, in advance as part of astatic configuration before use occurs, etc.

Another example of a type of use of target virtual network addresses isfor a migratable virtual network address that may be serially assignedto different computing nodes at different times. For example, amigratable target virtual network address may be assigned to aparticular service (e.g., provided by a particular application program,associated with providing access to a particular database, etc.) forwhich availability is desired, with a first computing node initiallyhaving primary ownership of the migratable target virtual networkaddress, but with the ability for one or more other second computingnodes to assume ownership of the migratable target virtual networkaddress (and provide the associated service) if the first computing nodebecomes unavailable. In some embodiments, such second computing nodes donot assert any association with the migratable target virtual networkaddress until the first computing node becomes unavailable, while inother embodiments at least one second computing node may specify orclaim a secondary association with the migratable target virtual networkaddress even while the first computing node is using the migratabletarget virtual network address (e.g., such as if the second computingnode is a live backup to the first computing node, and to enable a rapidswitch to that second computing node if the first computing node becomesunavailable). In addition, in some embodiments such dynamic changes ofownership or other association with a virtual network address areallowed only if that virtual network address has previously beenconfigured to be a migratable target virtual network address and/or onlyby computing nodes that have previously been configured to perform suchdynamic associations with a migratable target virtual network address,while in other embodiments such techniques may be used with any virtualnetwork address and/or by any computing node. Furthermore, as notedelsewhere, computing nodes of a managed computer network may dynamicallyassert or withdraw from association with such a migratable targetvirtual network address in at least some embodiments, including viarouting information included in routing communications such by thosecomputing nodes.

In addition, as previously noted, routing information received fromcomputing nodes of a managed computer network provided for a client mayin some embodiments be used to indicate one or more external publicnetwork addresses to be associated with the managed computer network,such as a range of public network addresses (e.g., an IP address prefix)that are globally routable over the Internet to enable external computersystems to communicate with the managed computer network. For example, aparticular client may have a range of public network addresses that theclient desires to use with the client's managed computer network. If so,a computing node of the managed computer network may, for example, usethe BGP routing protocol to announce those external public networkaddresses of the client within the managed computer network. When theconfigurable network service intercepts those routing communications, itdetermines to publicly announce those external public network addresses.As one example, the client may maintain a BGP infrastructure that isexternal to the managed computer network (e.g., at a remote privatenetwork of the client, such as a remote private network that may beconnected to the managed computer network via a VPN connection or otherconnection), and if so, the configurable network service may forwardinformation about the external public network addresses of the client(e.g., may forward the intercepted routing communications) to thatclient's existing infrastructure. In such a situation, the client'sexisting infrastructure performs the public announcement of the externalpublic network addresses, and communications from external computersystems to those external public network addresses may be routed overpublic networks to the client's existing infrastructure, which mayfurther forward those communications to the managed computer network(e.g., to the computing node of the managed computer network that issuedthe BGP routing communications) for further handling via one or moreconnections. In other embodiments, the configurable network service maypublicly announce such external public network addresses using its owncapabilities, such as via an edge communication manager module at a datacenter or other geographical location at which a managed computernetwork is located.

Furthermore, in at least some embodiments, before such external publicnetwork addresses are allowed to be used with a managed computer networkprovided to a client, the configurable network service may performvarious activities to verify the availability and appropriateness of theuse of those external public network addresses. For example, in somesuch embodiments, the client may need to provide configurationinformation for the managed computer network that includes a registeredAutonomous System Number, and optionally further demonstrate ownershipof those external public network addresses (or a non-ownership right touse those external public network addresses). Furthermore, in at leastsome embodiments, the client may be able to further configure routingfilters to be used for routes from the Internet, such as to restrictcommunications that are allowed to be received from and/or sent toparticular external computer systems. In other embodiments, theconfigurable network service may instead acquire various external publicnetwork addresses and make them available for use by clients in variousmanners, such as for a particular client to use one or more particularsuch external public network addresses (e.g., for a fee), for multipleclients to share use of one or more particular such external publicnetwork addresses (e.g., via PAT/NAT address translation techniques,such as may be used by an edge communication manager module as describedin greater detail elsewhere), etc.

Thus, the providing of networking functionality by a configurablenetwork service may include using routing information for managedvirtual computer networks in various manners. Various other aspects ofproviding managed computer networks are also described above, as well aselsewhere in this document, including to support virtual computernetworks that are overlaid on an underlying substrate network.Additional details are included below with respect to actions ofconfigurable network service embodiments to support networking devicesspecified for managed computer networks, such as for virtual networkingdevices that are not physically provided and whose functionality isemulated, as well as for obtaining and using routing information for amanaged computer network in various manners.

FIGS. 2A-2E illustrate further examples with additional illustrativedetails related to managing communications between computing nodes thatoccur via a virtual computer network overlaid over one or more physicalsubstrate networks, such as may be used by the computing nodes andnetworks of FIGS. 1A-1B, or in other situations. In these examples,FIGS. 2A and 2B illustrate details regarding actions of various modulesof an example CNS system in managing communications between computingnodes of a managed computer network, while FIGS. 2C and 2D illustrateadditional details regarding similar actions in managing communicationsin a manner specific to a particular specified network topology, such asthat includes virtual networking devices and/or intermediate computingnodes of the virtual computer network through which at least someinter-node communications pass. FIG. 2E illustrates additional detailsregarding obtaining and using routing information for a managed virtualcomputer network from routing-related communications directed to avirtual networking device of the managed virtual computer network.

In particular, FIG. 2A illustrates various example computing nodes 205and 255 that may communicate with each other by using one or moreintermediate interconnection networks 250 as a substrate network. Inthis example, the interconnection network 250 is an IPv6 substratenetwork on which IPv4 virtual computer networks are overlaid, althoughin other embodiments the interconnection network 250 and overlay virtualcomputer networks may use the same networking protocol (e.g., IPv4). Inaddition, in this example embodiment, the computing nodes are operatedon behalf of multiple distinct entities to whom managed virtual computernetworks are provided, and a System Manager module 290 manages theassociation of particular computing nodes with particular entities andmanaged virtual computer networks, and tracks various configurationinformation specified for the managed virtual computer networks. Theexample computing nodes of FIG. 2A include four computing nodes executedon behalf of an example entity Z and part of a corresponding managedvirtual computer network provided for entity Z, those being computingnodes 205 a, 205 c, 255 a and 255 b. In addition, other computing nodesare operated on behalf of other entities and belong to other providedvirtual computer networks, such as computing node 205 b and othercomputing nodes 255.

In this example, the computing nodes 205 are managed by and physicallyconnected to an associated Communication Manager module R 210, and thecomputing nodes 255 are managed by and physically connected to anassociated Communication Manager module S 260. The CNS CommunicationManager modules 210 and 260 are physically connected to aninterconnection network 250, as is the System Manager module 290,although the physical interconnections between computing nodes, modulesand the interconnection network are not illustrated in this example. Asone example, computing nodes 205 may each be one of multiple virtualmachines hosted by a single physical computing system, and CommunicationManager module R may be part of a hypervisor virtual machine monitor forthat physical computing system. For example, with reference to FIG. 1B,computing nodes 205 may represent the virtual machines 107 a, andcomputing nodes 255 may represent the virtual machines 107 d. If so,Communication Manager module R would correspond to Communication Managermodule 109 a of FIG. 1B, Communication Manager module S would correspondto Communication Manager module 109 d of FIG. 1B, the interconnectionnetwork 250 would correspond to interconnection network 122 of FIG. 1B,and the System Manager module 290 would correspond to System Managermodule 110 of FIG. 1B. Alternatively, computing nodes 205 or 255 mayinstead each be a distinct physical computing system, such as tocorrespond to computing systems 155 a-155 n of FIG. 1, or to computingnodes at other data centers or geographical locations (e.g., computingsystems at another data center 160, computing systems 145 a, etc.).

Each of the Communication Manager modules of FIG. 2A is associated witha group of multiple physical substrate network addresses, which theCommunication Manager modules manage on behalf of their associatedcomputing nodes. For example, Communication Manager module R is shown tobe associated with the IPv6 network address range of “::0A:01/72”, whichcorresponds to the 128-bit addresses (in hexadecimal) fromXXXX:XXXX:XXXX:XXXA:0100:0000:0000:0000 to XXXX:XXXX:XXXX:XXXA:01FF:FFFF:FFFF:FFFF (representing 2 to the power of 56 unique IPv6addresses), where each “X” may represent any hexadecimal character thatis appropriate for a particular situation. The interconnection network250 will forward any communication with a destination network address inthat range to Communication Manager module R—thus, with the initial 72bits of the range specified, the Communication Manager module R may usesubstrate network addresses that include the remaining available 56 bitsto represent the computing nodes that it manages and to determine how toprocess incoming communications whose destination network addresses arein that range.

For purposes of the example shown in FIG. 2A, computing nodes 205 a, 205c, 255 a, and 255 b are part of a single managed virtual computernetwork provided for entity Z, and have assigned IPv4 virtual networkaddresses for that virtual computer network of “10.0.0.2”, “10.0.5.1”,“10.0.0.3”, and “10.1.5.3”, respectively. Because computing node 205 bis part of a distinct managed virtual computer network that is providedfor entity Y, it can share the same virtual network address as computingnode 205 a without confusion. In this example, computing node A 205 aintends to communicate with computing node G 255 a, which are configuredin this example to be part of a single common physical sub-network (notshown) in a configured network topology for the managed virtual computernetwork, and the interconnection network 250 and Communication Managermodules are transparent to computing nodes A and G in this example. Inparticular, despite the physical separation of computing nodes A and G,the Communication Manager modules 210 and 260 operate so as to overlaythe managed virtual computer network for entity Z over the physicalinterconnection network 250 for communications between those computingnodes, including to emulate functionality corresponding to theconfigured sub-network of the managed virtual computer network, so thatthe lack of an actual corresponding network is transparent to thecomputing nodes A and G.

In order to send the communication to computing node G, computing node Aexchanges various messages 220 with Communication Manager module R 210,despite in the illustrated embodiment being unaware of the existence ofCommunication Manager module R (i.e., computing node A may believe thatit is transmitting a broadcast message to all other nodes on the localsub-network, such as via a specified switching device that computingnode A believes connects the nodes on the local sub-network). Inparticular, in this example, computing node A first sends an ARP messagerequest 220-a that includes the virtual network address for computingnode G (i.e., “10.0.0.3”) and that requests the corresponding hardwareaddress for computing node G (e.g., a 48-bit MAC address). CommunicationManager module R intercepts the ARP request 220-a, and responds tocomputing node A with a spoofed ARP response message 220-b that includesa virtual hardware address for computing node G.

To obtain the virtual hardware address for computing node G to use withthe response message, the Communication Manager module R first checks alocal store 212 of information that maps virtual hardware addresses tocorresponding IPv6 actual physical substrate network addresses, witheach of the virtual hardware addresses also corresponding to an IPv4virtual network address for a particular entity's managed virtualcomputer network. If the local store 212 does not contain an entry forcomputing node G (e.g., if none of the computing nodes 205 havepreviously communicated with computing node G and the System Managermodule 290 does not push mapping information to the CommunicationManager Module R without request, if a prior entry in local store 212for computing node G has expired based on an associated expiration time,etc.), the Communication Manager module R interacts 225 with SystemManager module 290 to obtain the corresponding actual IPv6 physicalsubstrate network address for computing node G on behalf of computingnode A. In particular, in this example, the System Manager module 290maintains provisioning information 292 that identifies where eachcomputing node is actually located and to which entity and/or managedvirtual computer network the computing node belongs, such as byinitiating execution of programs on computing nodes for entities andvirtual computer networks or by otherwise obtaining such provisioninginformation. As discussed in greater detail with respect to FIG. 2B, theSystem Manager module may determine whether the request fromCommunication Manager module R on behalf of computing node A forcomputing node G's actual IPv6 physical substrate network address isvalid, including whether computing node A is authorized to communicatewith computing node G (e.g., such as based on being part of the sameconfigured local area sub-network), and if so provides that actual IPv6physical substrate network address. While the interactions 225 with theSystem Manager module 290 are not illustrated in this example asexplicitly traveling over the interconnection network 250, it will beappreciated that such interactions may include one or morecommunications that do travel over the interconnection network 250 in atleast some embodiments.

Communication Manager module R receives the actual IPv6 physicalsubstrate network address for computing node G from the System Managermodule 290, and stores this received information as part of an entry forcomputing node G in mapping information 212 for later use (optionallywith an expiration time and/or other information). In addition, in thisexample, Communication Manager module R determines a virtual hardwareaddress to be used for computing node G (e.g., by generating a dummyidentifier that is locally unique for the computing nodes managed byCommunication Manager module R), stores that virtual hardware address inconjunction with the received actual IPv6 physical substrate networkaddress as part of the mapping information entry, and provides thevirtual hardware address to computing node A as part of response message220-b. By maintaining such mapping information 212, later communicationsfrom computing node A to computing node G may be authorized byCommunication Manager module R without further interactions with theSystem Manager module 290, based on the use of the virtual hardwareaddress previously provided by Communication Manager module R. In someembodiments, the hardware address used by Communication Manager module Rfor computing node G may not be a dummy address, such as if SystemManager module 290 further maintains information about hardwareaddresses used by the various computing nodes (e.g., virtual hardwareaddresses assigned to virtual machine computing nodes, actual hardwareaddresses assigned to physical computing systems acting as computingnodes, etc.) and provides the hardware address used by computing node Gto Communication Manager module R as part of the interactions 225. Insuch embodiments, the Communication Manager module R may take furtheractions if computing nodes on different virtual networks use the samevirtual hardware address, such as to map each combination of computingnode hardware address and managed virtual computer network to acorresponding substrate network address.

In other embodiments, Communication Manager module R may interact withSystem Manager module 290 to obtain a physical substrate network addressfor computing node G or otherwise determine such a physical substratenetwork address at times other than upon receiving an ARP request, suchas in response to any received communication that is directed tocomputing node G using the virtual network address “10.0.0.3” as part ofentity Z's virtual computer network, or if the System Manager moduleprovides that information to Communication Manager module R withoutrequest (e.g., periodically; upon changes in the information, such aschanges in network topology information for the managed virtual computernetwork; etc.). Furthermore, in other embodiments, the virtual hardwareaddresses that are used may differ from this example, such as if thevirtual hardware addresses are specified by the System Manager module290, if the virtual hardware addresses are not random and instead storeone or more types of information specific to the corresponding computingnodes, etc. In addition, in this example, if computing node A had notbeen determined to be authorized to send communications to computingnode G, whether by the System Manager module 290 and/or CommunicationManager module R, Communication Manager module R would not send theresponse message 220-b with the virtual hardware address (e.g., insteadsends no response or an error message response).

In this example, the returned IPv6 actual physical substrate networkaddress corresponding to computing node G in interactions 225 is“::06:02:<Z-identifier>:10.0.0.3”, where “10.0.0.3” is stored in thelast 32 bits of the 128-bit IPv6 address, and where “<Z-identifier>” isa 24-bit entity network identifier for computing node G corresponding tothe managed virtual computer network for entity Z (e.g., as previouslyassigned by the System Manager module to that network to reflect arandom number or some other number corresponding to the entity). Theinitial 72 bits of the IPv6 network address store the “::0B:02”designation, corresponding to the sub-network or other portion of thephysical interconnection network with a network address range of“::0B:02/72” to which Communication Manager module S corresponds—thus, acommunication sent over the interconnection network 250 to IPv6destination network address “::0B:02:<Z-identifier>:10.0.0.3” will berouted to Communication Manager module S. In other embodiments, theentity network identifier may be other lengths (e.g., 32 bits, ifCommunication Manager module S has an associated network address rangeof 64 bits rather than 56 bits) and/or may have other forms (e.g., maybe random, may store various types of information, etc.), and theremaining 56 bits used for the network address range after the “::0B:02”designation may store other types of information (e.g., an identifierfor a particular entity, a tag or label for the virtual computernetwork, an identifier for a particular specified VLAN to whichcomputing node G is assigned, etc.). In other embodiments, some or allsuch information may instead be stored and/or transmitted with acommunication to computing node G in other manners, such as by includingthe information in a header of the communication, including insituations in which the substrate network uses the IPv4 networkingprotocol.

After receiving the response message 220-b from Communication Managermodule R, computing node A creates and initiates the sending of aninter-node communication to computing node G, shown in FIG. 2A ascommunication 220-c. In particular, the header of communication 220-cincludes a destination virtual network address for destination computingnode G that is “10.0.0.3”, a destination hardware address fordestination computing node G that is the virtual hardware addressprovided to computing node A in message 220-b, a source virtual networkaddress for sending computing node A that is “10.0.0.2”, and a sourcehardware address for sending computing node A that is an actual or dummyhardware address that was previously identified to computing node A(e.g., by Communication Manager module R, based on prior configurationof computing node A, etc.). Since computing node A believes thatcomputing node G is part of the same local sub-network as itself,computing node A does not need to direct the communication 220-c to anyintermediate virtual router devices that are configured in a networktopology for the managed virtual computer network to separate thecomputing nodes. Furthermore, in this example, there are no othercomputing nodes of the managed virtual computer network through whichthis inter-node communication is configured to pass.

Communication Manager module R intercepts the communication 220-c,modifies the communication as appropriate, and forwards the modifiedcommunication over the interconnection network 250 to computing node G.In particular, Communication Manager module R extracts the virtualdestination network address and virtual destination hardware address forcomputing node G from the header, and then retrieves the IPv6 actualphysical substrate network address corresponding to that virtualdestination hardware address from mapping information 212. As previouslynoted, the IPv6 actual physical substrate network address in thisexample is “::0B:02:<Z-identifier>:10.0.0.3”, and Communication Managermodule R creates a new IPv6 header that includes that actual physicalsubstrate network address as the destination address. Similarly, theCommunication Manager module R extracts the virtual source networkaddress and virtual source hardware address for computing node A fromthe header of the received communication, obtains an IPv6 actualphysical substrate network address corresponding to that virtual sourcehardware address (e.g., from a stored entry in mapping information 212,by interacting with the System Manager module 290 to obtain thatinformation if not previously obtained, etc.), and includes that actualphysical substrate network address as the source network address for thenew IPv6 header. In this example, the IPv6 actual physical substratenetwork address for computing node A is“::0A:01:<Z-identifier>:10.0.0.2”, which if used in a reply byCommunication Manager module S on behalf of computing node G will berouted to Communication Manager module R for forwarding to computingnode A. The Communication Manager module R then creates communication230-3 by modifying communication 220-c so as to replace the prior IPv4header with the new IPv6 header (e.g., in accordance with SIIT),including populating the new IPv6 header with other information asappropriate for the communication (e.g., payload length, traffic classpacket priority, etc.). Thus, the communication 230-3 includes the samecontent or payload as communication 220-c, without encapsulating thecommunication 220-c within the communication 230-3 in this example.Furthermore, access to the specific information within the payload isnot needed for such re-headering, such as to allow Communication Managermodule R to handle communications in which the payload is encryptedwithout needing to decrypt that payload.

In at least some embodiments, before forwarding communication 230-3 to

Communication Manager module S, Communication Manager module R mayperform one or more actions to determine that communication 220-c isauthorized to be forwarded to computing node G as communication 230-3,such as based on the mapping information 212 including a valid entry forthe destination virtual hardware address used in communication 220-c(e.g., an entry specific to sending computing node 205 a in someembodiments, or instead an entry corresponding to any of the computingnodes 205 in other embodiments). In other embodiments, such anauthorization determination may not be performed by CommunicationManager module R for each outgoing communication, or instead may beperformed in other manners (e.g., based on a determination that thesending node and destination node are part of the same managed virtualcomputer network, are associated with the same entity, or are otherwiseauthorized to inter-communicate; based on an interaction with SystemManager module 290 to obtain an authorization determination for thecommunication; etc.).

After Communication Manager module R forwards the modified communication230-3 to the interconnection network 250, the interconnection networkuses the physical IPv6 destination network address of the communicationto route the communication to Communication Manager module S. In doingso, the devices of the interconnection network 250 do not use theportion of the destination network address that includes the embeddedentity network identifier or embedded virtual network address, and thusdo not need any special configuration to forward such a communication,nor even awareness that a managed virtual computer network is beingoverlaid on the physical interconnection network.

When Communication Manager module S receives communication 230-3 via theinterconnection network 250, it performs actions similar to those ofCommunication Manager module R, but in reverse. In particular, in atleast some embodiments, the Communication Manager module S verifies thatcommunication 230-3 is legitimate and authorized to be forwarded tocomputing node G, such as via one or more interactions 240 with theSystem Manager module. As with the interactions 225, it will beappreciated that the interactions 240 may include one or morecommunications (not shown) that travel over the interconnection network250 in at least some embodiments. If the communication 230-3 isdetermined to be authorized (or if the authorization determination isnot performed), the Communication Manager module S then modifiescommunication 230-3 as appropriate and forwards the modifiedcommunication to computing node G. Additional details related to theverification of the communication 230-3 are discussed with respect toFIG. 2B.

In particular, to modify communication 230-3, Communication Managermodule S retrieves information from mapping information 262 thatcorresponds to computing node G, including the virtual hardware addressused by computing node G (or generates such a virtual hardware addressif not previously available, such as for a new computing node).Communication Manager module S then creates communication 245-e bymodifying communication 230-3 so as to replace the prior IPv6 headerwith a new IPv4 header (e.g., in accordance with SIIT). The new IPv4header includes the virtual network address and virtual hardware addressfor computing node G as the destination network address and destinationhardware address for the new IPv4 header, the virtual network addressand a virtual hardware address for computing node A as the sourcenetwork address and source hardware address for the new IPv4 header, andincludes other information as appropriate for the communication (e.g.,total length, header checksum, etc.). The virtual hardware address usedby Communication Manager module S for computing node A may be the sameas the hardware address used by Communication Manager module R forcomputing node A, but in other embodiments each Communication Managermodule may maintain separate hardware address information that is notrelated to the information used by the other Communication Managermodules (e.g., if Communication Manager module S generated its own dummyvirtual hardware address for computing node A in response to a prior ARPrequest from one of the computing nodes 255 for computing node A'shardware address). Thus, the communication 245-e includes the samecontent or payload as communications 220-c and 230-3. CommunicationManager module S then forwards communication 245-e to computing node G.

After receiving communication 245-e, computing node G determines to senda response communication 245-f to computing node A, using the sourcevirtual network address and source virtual hardware address forcomputing node A from communication 245-e. Communication Manager moduleS receives response communication 245-f, and processes it in a mannersimilar to that previously described with respect to communication 220-cand Communication Manager module R. In particular, Communication Managermodule S optionally verifies that computing node G is authorized to sendcommunications to computing node A (e.g., based on being a response to aprevious communication, or otherwise based on configuration informationfor computing nodes A and G as previously described), and then modifiescommunication 245-f to create communication 230-6 by generating a newIPv6 header using mapping information 262. After forwardingcommunication 230-6 to the interconnection network 250, thecommunication is sent to Communication Manager module R, which processesthe incoming communication in a manner similar to that previouslydescribed with respect to communication 230-3 and Communication Managermodule S. In particular, Communication Manager module R optionallyverifies that computing node G is authorized to send communications tocomputing node A and that communication 230-6 actually was sent from thesubstrate network location of computing node G, and then modifiescommunication 230-6 to create response communication 220-d by generatinga new IPv4 header using mapping information 212. Communication Managermodule R then forwards response communication 220-d to computing node A.In some embodiments and situations, Communication Manager modules Rand/or S may handle response communications differently from initialcommunications, such as to assume that response communications areauthorized in at least some situations, and to not perform some or allauthorization activities for response communications in thosesituations.

In this manner, computing nodes A and G may inter-communicate using aIPv4-based managed virtual computer network, without any specialconfiguration of those computing nodes to handle the actual interveningIPv6-based substrate interconnection network, and interconnectionnetwork 250 may forward IPv6 communications without any specialconfiguration of any physical networking devices of the interconnectionnetwork, based on the Communication Manager modules overlaying thevirtual computer network over the actual physical interconnectionnetwork from the edges of the interconnection network.

In addition, while not illustrated with respect to FIG. 2A, in at leastsome embodiments the Communication Manager modules may receive andhandle other types of requests and communications on behalf ofassociated computing nodes. For example, Communication Manager modulesmay take various actions to support broadcast and multicast capabilitiesfor computing nodes that they manage. As one example, in someembodiments, a special multicast group virtual network address suffixmay be reserved from each entity network identifier prefix for use insignaling networking Layer 2 raw encapsulated communications. Similarly,for link-local broadcast and multicast communications, a specialmulticast group /64 prefix may be reserved (e.g., “FF36:0000::”), whilea different destination address prefix (e.g., “FF15:0000::”) may be usedfor other multicast communications. Thus, for example, multicast andbroadcast IP frames may be encapsulated using a corresponding reserved64-bit prefix for the first 64 bits of the 128-bit IPv6 address, withthe remaining 64 bits including the virtual IPv4 network address for thedestination computing node and the entity network identifier for thedestination computing node in a manner similar to that previouslydescribed. Alternatively, in other embodiments, one or more types ofbroadcast and/or multicast communications may each have a correspondingreserved label or other identifier that has a different value or form,including using a different number of bits and/or being stored in amanner other than as a network address prefix. When a computing nodesends a broadcast/multicast communication, any Communication Managermodule with an associated computing node that has subscribed to thatmulticast/broadcast group would be identified (e.g., based on thoseCommunication Manager modules having subscribed to the group, such as inresponse to prior join communications sent by those associated computingnodes), and the Communication Manager module for the sending computingnode would forward the communication to each of the identifiedCommunication Manager modules of the group, for forwarding to theirappropriate managed computing nodes. In addition, in some embodimentsand situations, at least some broadcast or multicast communications maynot be forwarded by Communication Manager modules, such ascommunications with an IPv4 prefix of 224.0/16 or another designatedprefix or other label or identifier.

In addition to supporting broadcast and multicast capabilities formanaged computing nodes, the Communication Manager modules may receiveand handle other types of requests and communications on behalf ofassociated computing nodes that correspond to configured networktopologies for the virtual computer networks to which the computingnodes belong. For example, computing nodes may send various requeststhat a specified local router device or other specified networkingdevice would be expected to handle (e.g., ping requests, SNMP queries,etc.), and the associated Communication Manager modules may interceptsuch requests and take various corresponding actions to emulate thefunctionality that would have been provided by the specified networkingdevice if it was physically implemented. Furthermore, in at least someembodiments, Communication Manager modules may receive and handlerouting-related communications that are directed to virtual networkingdevices rather than to computing nodes of the managed virtual computernetwork, as discussed in greater detail with respect to FIG. 2E.

In addition, it will be appreciated that a Communication Manager modulemay facilitate communications between multiple of the computing nodesthat are associated with that Communication Manager module. For example,with respect to FIG. 2A, computing node 205 a may wish to send anadditional communication (not shown) to computing node 205 c. If so,Communication Manager module R would perform actions similar to thosepreviously described with respect to the handling of outgoingcommunication 220-c by Communication Manager module R and the handlingof incoming communication 230-3 by Communication Manager module S, butwithout re-headering of the additional communication to use an IPv6header (since the communication will not travel over the interconnectionnetwork). However, if computing nodes 205 a and 205 c are configured ina network topology for the virtual computer network to be separated byone or more virtual networking devices, the Communication Manager moduleR may take additional actions to emulate the functionality of thosevirtual networking devices, as discussed in greater detail with respectto FIG. 2C. Similarly, if computing nodes 205 a and 205 c are configuredin a network topology for the virtual computer network to be separatedby one or more other intermediate computing nodes of the virtualcomputer network through which such communications are to pass, theCommunication Manager module R may take additional actions to forwardsuch communications to such intermediate computing nodes, as discussedin greater detail with respect to FIG. 2D.

While not illustrated with respect to FIG. 2A, in at least someembodiments other types of requests and communications may also behandled in various ways. For example, in at least some embodiments, anentity may have one or more computing nodes that are managed byCommunication Manager module(s) and that are part of a managed virtualcomputer network for that entity, and may further have one or more othernon-managed computing systems (e.g., computing systems that are directlyconnected to the interconnection network 250 and/or that natively useIPv6 network addressing) that do not have an associated CommunicationManager module that manages their communications. If the entity desiresthat those non-managed computing systems be part of that virtualcomputer network or otherwise communicate with the managed computingnodes of the virtual computer network, such communications betweenmanaged computing nodes and non-managed computing systems may be handledby the Communication Manager module(s) that manage the one or morecomputing nodes in at least some such embodiments. For example, in suchsituations, if such a non-managed computing system is provided with anactual IPv6 destination network address for such a managed computingnode (e.g., “::0A:01:<Z-identifier>:10.0.0.2” for managed computing nodeA in this example), the non-managed computing system may sendcommunications to computing node A via interconnection network 250 usingthat destination network address, and Communication Manager module Rwould forward those communications to computing node A (e.g., afterre-headering the communications in a manner similar to that previouslydescribed) if Communication Manager module R is configured to acceptcommunications from that non-managed computing system (or from anynon-managed computing system). Furthermore, Communication Manager moduleR may generate a dummy virtual network address to correspond to such anon-managed computing system, map it to the actual IPv6 network addressfor the non-managed computing system, and provide the dummy virtualnetwork address to computing node A (e.g., as the source network addressfor the communications forwarded to computing node A from thenon-managed computing system), thus allowing computing node A to sendcommunications to the non-managed computing system.

Similarly, in at least some embodiments and situations, at least somemanaged computing nodes and/or their virtual computer networks may beconfigured to allow communications with other devices that are not partof the virtual computer network, such as other non-managed computingsystems or other types of network appliance devices that do not have anassociated Communication Manager module that manages theircommunications. In such situations, if the managed computing nodesand/or the virtual computer network is configured to allowcommunications with such other non-managed devices, such a non-manageddevice may similarly be provided with the actual IPv6 destinationnetwork address for such a computing node (e.g.,“::0A:01:<Z-identifier>:10.0.0.2” for computing node A in this example),allowing the non-managed device to send communications to computing nodeA via interconnection network 250 using that destination networkaddress, with Communication Manager module R then forwarding thosecommunications to computing node A (e.g., after re-headering thecommunications in a manner similar to that previously described).Furthermore, Communication Manager module R may similarly manageoutgoing communications from computing node A to such a non-manageddevice to allow computing node A to send such communications.

In addition, as previously noted, a communication manager module managescommunications for associated computing nodes in various ways, includingin some embodiments by assigning virtual network addresses to computingnodes of a provided virtual computer network, and/or by assigningsubstrate physical network addresses to managed computing nodes from arange of substrate physical network addresses that correspond to thecommunication manager module. In other embodiments, some such activitiesmay instead be performed by one or more computing nodes of the virtualcomputer network, such as to allow a DHCP (Dynamic Host ConfigurationProtocol) server or other device of a virtual computer network tospecify virtual network addresses and/or substrate physical networkaddresses for particular computing nodes of the virtual network. In suchembodiments, the communication manager module obtains such configurationinformation from the virtual network device(s), and updates its mappinginformation accordingly (and in some embodiments may further update oneor more system manager modules that maintain information about computingnodes associated with virtual networks). In yet other embodiments, auser or other entity associated with a virtual computer network maydirectly configure particular computing nodes to use particular virtualnetwork addresses. If so, the communication manager modules and/orsystem manager module may track which virtual network addresses are usedby particular computing nodes, and similarly update stored mappinginformation accordingly.

In addition, in some embodiments and situations, a managed computingnode may itself be treated as a phantom router, with multiple virtualnetwork addresses associated with that managed computing node, and withthat managed computing node forwarding communications to other computingnodes that correspond to those multiple virtual network addresses. Insuch embodiments, the communication manager module that managescommunications for that managed router computing node handlescommunications to and from that computing node in a manner similar tothat previously described. However, the communication manager module isconfigured with the multiple virtual network addresses that correspondto the managed router computing node, so that incoming communications toany of those multiple virtual network addresses are forwarded to themanaged router computing node, and so that outgoing communications fromthe managed router computing node are given a substrate source physicalnetwork address that corresponds to the particular computing node thatsent the communication via the managed router computing node. In thismanner, routers or other networking devices of a particular customer orother entity may be virtually represented for a virtual computer networkimplemented for that entity.

FIG. 2B illustrates some of the computing nodes and communicationsdiscussed with respect to FIG. 2A, but provides additional details withrespect to some actions taken by the Communication Manager modules 210and 260 and/or the System Manager module 290 to authorize communicationsbetween computing nodes. For example, after computing node A sendsmessage 220-a to request a hardware address for computing node G,Communication Manager module R may perform one or more interactions 225with the System Manager module 290 in order to determine whether toprovide that information, such as based on whether computing node A isauthorized to communicate with computing node G, as well as to determinea corresponding substrate physical network address for computing node Gbased on interconnection network 250. If the Communication Managermodule R has previously obtained and stored that information and itremains valid (e.g., has not expired), then the interactions 225 may notbe performed. In this example, to obtain the desired physical networkaddress corresponding to computing node G, Communication Manager moduleR sends a message 225-1 to the System Manager module 290 that includesthe virtual network addresses for computing nodes A and G, and thatincludes an entity network identifier for each of the computing nodes,which in this example is an entity network identifier for the managedvirtual computer network of entity Z (e.g., a 32-bit or 24-bit uniqueidentifier). In at least some embodiments, Communication Manager moduleR may send message 225-1 to the System Manager module 290 using ananycast addressing and routing scheme, so that multiple System Managermodules (not shown) may be implemented (e.g., one for each data centerthat includes Communication Manager modules and associated computingnodes) and an appropriate one of those (e.g., the nearest, the mostunderutilized, etc.) is selected to receive and handle the message.

After the System Manager module 290 determines that computing node A isauthorized to communicate with computing node G (e.g., based on havingthe same entity network identifier, based on computing node A having anentity network identifier that is authorized to communicate withcomputing nodes of the entity network identifier for computing node G,based on other information provided by or associated with computing nodeA indicating that computing node A is authorized to perform suchcommunications, based on information provided by or associated withcomputing node G indicating that computing node A is authorized toperform such communications, based on the configured network topologyfor the managed virtual computer network to which computing nodes A andG belong, etc.), the System Manager module 290 returns a responsemessage 225-2 that includes the desired actual physical substratenetwork address corresponding to computing node G. In addition, in atleast some embodiments, before sending the desired actual physicalnetwork address, the System Manager module 290 may further verify thatCommunication Manager module R is authorized to send the message 225-1on behalf of computing node A, such as based on computing node A beingdetermined to be one of the computing nodes with which CommunicationManager module R is associated.

In other embodiments, Communication Manager module R may perform some orall of the actions described as being performed by System Manager module290, such as to maintain provisioning information for the variouscomputing nodes and/or to determine whether computing node A isauthorized to send communications to computing node G, or instead nosuch authorization determination may be performed in some or allsituations. Furthermore, in other embodiments, other types ofauthorization determinations may be performed for a communicationbetween two or more computing nodes, such as based on a type of thecommunication, on a size of the communication, on a time of thecommunication, etc.

As previously noted with respect to FIG. 2A, after Communication Managermodule S receives communication 230-3 intended for computing node G viathe interconnection network 250, Communication Manager module S mayperform one or more interactions 240 with the System Manager module 290in order to determine whether to authorize that communication. Inparticular, in this example, to verify that the communication 230-3 isvalid and authorized to be forwarded to computing node G, CommunicationManager module S first extracts the actual IPv6 destination networkaddress and actual IPv6 source network address from the header ofcommunication 230-3, and then retrieves the embedded entity networkidentifiers and virtual network addresses from each of the extractedIPv6 network addresses. The Communication Manager module S nextexchanges messages 240 with System Manager module 290 to obtain thecorresponding actual IPv6 physical network address for the sendingcomputing node A on behalf of computing node G, including a message240-4 that includes the extracted virtual network addresses forcomputing nodes A and G and the entity network identifier for each ofthe computing nodes. In at least some embodiments, Communication Managermodule S may send message 240-4 to the System Manager module 290 usingan anycast addressing and routing scheme, as previously described.

The System Manager module 290 receives message 240-4, and returns aresponse message 240-5 that includes the actual physical substratenetwork address corresponding to computing node A, which in this exampleis “::0A:01:<Z-identifier>:10.0.0.2”. As previously discussed withrespect to messages 225-1 and 225-2, in some embodiments the SystemManager module 290 and/or Communication Manager module S may furtherperform one or more other types of authorization determinationactivities, such as to determine that computing node G is authorized tocommunicate with computing node A, that Communication Manager module Sis authorized to send the message 240-4 on behalf of computing node G,etc. Communication Manager module S then verifies that the returnedphysical network address in response message 240-5 matches the sourceIPv6 network address extracted from the header of communication 230-3,so as to prevent attempts to spoof messages as being from computing nodeA that are actually sent from other computing nodes in other locations.Communication Manager module S optionally stores this receivedinformation from response message 240-5 as part of an entry forcomputing node A in mapping information 262 for later use, along withcomputing node A's virtual network address and a virtual hardwareaddress for computing node A.

FIG. 2C illustrates a further example of managing ongoing communicationsfor the virtual computer network described with respect to FIGS. 2A and2B, but with communications being managed to support virtual networkingfunctionality for the virtual computer network in accordance with aconfigured network topology for the virtual computer network. Thenetwork topology information may be configured for the managed virtualcomputer network in various manners and at various times. For example,entity Z 285 may directly interact with the System Manager module 290 toprovide the configured network topology information, such as via an API(not shown) and/or GUI (not shown) that is provided by the module 290 tofacilitate obtaining such configuration information. In addition, thenetwork topology information may be configured, for example, as part ofinitially creating the virtual computer network for entity Z, and insome embodiments may further be dynamically updated while the virtualcomputer network is in use.

The example of FIG. 2C illustrates computing node A, Communication

Manager modules R and S, System Manager module 290, and interconnectionnetwork 250 in a manner similar to that shown in FIGS. 2A and 2B.However, FIG. 2C further illustrates additional information regardingcomputing node A 205 a and computing node H 255 b as compared to FIG.2A, as well as logical representations 270 a and 270 b of two specifiedrouter devices that are part of the configured network topology for themanaged virtual computer network but that are not actually physicallyimplemented as part of providing the managed virtual computer network.In particular, in this example, computing node A is sending acommunication to computing node H, and the actions of the physicallyimplemented modules 210 and 260 and devices of network 250 in actuallysending the communication are shown, as well as emulated actions of thevirtual router devices 270 a and 270 b in logically sending thecommunication.

In this example, computing nodes A and H are configured to be part oftwo distinct sub-networks of the virtual computer network, and thevirtual router devices 270 a and 270 b separate the computing nodes Aand H in the configured network topology for the virtual computernetwork. For example, virtual router device J 270 a may be a localrouter device to computing node A in the configured network topology(e.g., may manage a first sub-network that includes computing node A),and virtual router device L 270 b may be a local router device tocomputing node H in the configured network topology (e.g., may manage adistinct second sub-network that includes computing node H). Whilecomputing nodes A and H are illustrated as being separated by two routerdevices in the configured network topology in this example, it will beappreciated that two such computing nodes may be separated by 0, 1 ormore than 2 router devices in other situations, and that other types ofnetworking devices may separate computing nodes in some situations.

In the example of FIG. 2C, the additional information that is shown forcomputing nodes A and H includes hardware addresses associated withthose computing nodes for the virtual computer network, such as virtualhardware addresses that are assigned to the computing nodes by theSystem Manager module 290 and/or the Communication Manager modules R andS. In particular, in this example, computing node A has been assignedhardware address “00-05-02-0B-27-44,” and computing node H has beenassigned hardware address “00-00-7D-A2-34-11.” In addition, the virtualrouter devices J and L have also each been assigned virtual hardwareaddresses, which in this example are “00-01-42-09-88-73” and“00-01-42-CD-11-01,” respectively, and have been assigned virtualnetwork addresses, which in this example are “10.0.0.1” and “10.1.5.1,”respectively. The various hardware addresses will be used as part of thesending of the communication from computing node A to computing node H,and the providing of corresponding virtual networking functionality forthe virtual computer network, as described below.

Thus, in a manner similar to that described with respect to FIG. 2A,computing node A determines to send a communication to computing node H,and accordingly exchanges various messages 222 with CommunicationManager module R 210. In particular, in this example, computing node Afirst sends an ARP message request 222-a for virtual hardware addressinformation. However, unlike the example of FIG. 2A in which computingnodes A and G were part of the same logical sub-network, communicationsfrom computing node A to computing node H are expected to first passthrough an initial intermediate destination of local router device Jbefore being forwarded to computing node H. Accordingly, since virtualrouter J is the initial intermediate destination for logically remotecomputing node H, the ARP message request 222-a includes the virtualnetwork address for virtual router J (i.e., “10.0.0.1”) and requests thecorresponding hardware address for virtual router J. In otherembodiments, computing node A may instead request virtual hardwareaddress information for computing node H directly (e.g., using thevirtual network address “10.1.5.3” for computing node H), but beprovided with the corresponding hardware address for virtual router J.

Communication Manager module R intercepts the ARP request 222-a, andobtains a hardware address to provide to computing node A as part ofspoofed ARP response message 222-b. The Communication Manager module Rmay determine the hardware address for virtual router J, as well as thatcomputing node H is part of a distinct logical sub-network fromcomputing node A, in various manners in various embodiments. Forexample, as previously discussed, the Communication Manager module R maystore various hardware address information as part of mappinginformation 212 c, and if so may already have stored hardware addressinformation for virtual router J and/or substrate network addressinformation for computing node H. If not, however, Communication Managermodule R performs one or more interactions 227 with the System Managermodule 290 to obtain information from the module 290 corresponding tothe indicated virtual network address for virtual router J. Rather thanobtaining a substrate network address corresponding to the indicatedvirtual network address, as for computing node G in FIG. 2A, the SystemManager module 290 indicates that the virtual network addresscorresponds to a virtual router device of the configured networktopology, and supplies a substrate network address that corresponds tothe final destination computing node H, and may also provide thehardware address information for virtual router J (e.g., by using theprovisioning information 292 and/or the network topology information296). In particular, the System Manager module 290 maintains variousinformation 296 related to the configured network topology for thevirtual computer networks that it provides or otherwise manages, such asinformation about specified networking devices, and uses thatinformation and the provisioning information 292 to provide requestedinformation to Communication Manager modules. The information 296 may insome embodiments be generated at least in part by a separate NetworkRouting Manager module (not shown), as described in greater detail withrespect to FIG. 2E and elsewhere. The Communication Manager module Rthen stores the received information as part of mapping information 212c for future use, and in this manner determines that computing node H ispart of a distinct sub-network from computing node A in the configurednetwork topology. Furthermore, Communication Manager module R providescomputing node A with the hardware address “00-01-42-09-88-73”corresponding to virtual router J as part of response message 222-b.While request 222-a and response message 222-b actually pass betweencomputing node A and Communication Manager module R in the mannerdiscussed, from the standpoint of computing node A, the communications222-a and 222-b are part of logical interactions 263 that occur withlocal router device J.

After receiving the response message 222-b from Communication Managermodule R, computing node A creates and initiates the sending of acommunication to computing node H, shown in FIG. 2C as communication222-c. In particular, the header of communication 222-c includes adestination network address for destination computing node H that is“10.1.5.3”, a destination hardware address that is the virtual hardwareaddress for virtual router J provided to computing node A in message222-b, a source network address for sending computing node A that is“10.0.0.2”, and a source hardware address for sending computing node Athat is an actual or dummy hardware address that was previouslyidentified to computing node A. From the standpoint of computing node A,the sent communication will be handled in the manner illustrated forlogical communication 265, and will be sent to local virtual router J ascommunication 265 a for forwarding based on the destination hardwareaddress in the communication. If virtual router J were physicallyimplemented and received such a communication 265 a, it would modify theheader of the communication 265 a and forward the modified communication265 b to virtual router L, which would similarly modify the header ofthe communication 265 b and forward that further modified communication265 c to computing node H. The modifications that virtual router J wouldperform to such a communication 265 a may include, for example,decrementing a TTL network hop value in the header and changing thedestination hardware address to correspond to the next destination,which in this example would be virtual router L. Similarly, themodifications that virtual router L would perform to such acommunication 265 b may include, for example, further decrementing theTTL network hop value in the header and changing the destinationhardware address to correspond to the next destination, which in thisexample would be computing node H. In some embodiments and situations,other similar modifications may be performed by the router devices ifthey were physically implemented and used for the forwarding of thecommunication.

While communication 222-c from computing node A to computing node H islogically handled in the manner illustrated for communication 265, thecommunication 222-c is actually intercepted and handled by CommunicationManager module R. In particular, in a manner similar to that describedin FIG. 2A for communication 220-c, Communication Manager module Rintercepts the communication 222-c, modifies the communication asappropriate, and forwards the modified communication over theinterconnection network 250 to computing node H. To determine thesubstrate network address to be used for forwarding the modifiedcommunication over the interconnection network 250, CommunicationManager module R extracts the destination virtual network address anddestination virtual hardware address from the header of communication222-c. However, based on the destination virtual hardware addresscorresponding to virtual router J, Communication Manager module Rdetermines to use the destination virtual network address to identifythe destination substrate network address, in a manner different fromthat of FIG. 2A. Thus, the Communication Manager module R checks themapping information 212 c to determine if a substrate network addresscorresponding to computing node H's virtual network address has beenpreviously determined and stored. If not, Communication Manager module Rperforms one or more interactions 227 with the System Manager module 290to determine that information, in a manner similar to the interactions225 of FIG. 2A. As discussed in greater detail with respect to FIG. 2B,in response to the ARP request message 222-a and/or communication 222-c,the Communication Manager module R and/or the System Manager module 290may further perform various optional authentication activities.

After Communication Manager module R determines the IPv6 actual physicalsubstrate network address corresponding to computing node H, it createsa new IPv6 header that includes that actual physical substrate networkaddress as the destination address, and similarly adds a source IPv6address for computing node A to the new header. In this example, thephysical substrate network address corresponding to computing node H issimilar to that of computing node G, and in particular is the IPv6substrate network address “::06:02:<Z-identifier>:10.1.5.3”, where“10.1.5.3” is stored in the last 32 bits of the 128-bit IPv6 address,and where “<Z-identifier>” is a 24-bit entity network identifier for themanaged virtual computer network. Thus, as with communications sent tocomputing node G, a communication sent over the interconnection network250 to the substrate network address for computing node H will be routedto Communication Manager module S. The Communication Manager module Rnext creates a new communication 232-3 by modifying communication 222-cso as to replace the prior IPv4 header with the new IPv6 header (e.g.,in accordance with SIIT), including populating the new IPv6 header withother information as appropriate for the new communication (e.g.,payload length, traffic class packet priority, etc.), and forwardscommunication 232-3 over the interconnection network 250. Theinterconnection network then uses the physical IPv6 destination networkaddress of the communication 232-3 to route the communication toCommunication Manager module S. When Communication Manager module Sreceives communication 232-3 via the interconnection network 250, itperforms actions similar to those described in FIG. 2A with respect tocommunication 230-3, including to optionally perform interactions 242with the System Manager module 290 to determine if the communication isauthorized, to update mapping information 262 c to reflect any newinformation about computing node A, to modify the communication toinclude an appropriate IPv4 header, and to provide the modifiedcommunication as communication 247-e to computing node H.

Furthermore, as noted elsewhere, Communication Manager module R and/orCommunication Manager module S take further actions in this example tomodify the communication from computing node A to computing node H insuch a manner as to provide virtual networking functionalitycorresponding to the configured network topology for the virtualcomputer network, including to emulate functionality that would beprovided by virtual routers J and L if they were physically implementedfor the virtual computer network. For example, as previously discussed,virtual routers J and L would perform various modifications tocommunication 265 as it is forwarded to computing node H if thoserouters were physically implemented and used, including to modify TTLnetwork hop values and to perform other header modifications.Accordingly, Communication Manager module R and/or Communication Managermodule S may perform similar modifications to the communication 222-cand/or 247-e to emulate such functionality of the virtual routers J andL. Thus, computing node H receives a communication 247-e that appears tobe communication 265 c forwarded via the specified network topology forthe virtual computer network.

In this manner, the CNS system may provide virtual networkingfunctionality corresponding to the configured network topology, withoutany special configuration of the computing nodes of the managed virtualcomputer network or of the physical networking devices of theintervening substrate interconnection network, based on theCommunication Manager modules overlaying the virtual computer network onthe actual physical interconnection network in such a manner as toemulate the configured network topology. In addition, multiple modulesof the CNS system may operate together in a distributed manner toprovide functionality corresponding to a particular virtual networkingdevice, such as with modules 210, 260 and 290 operating together in theprevious example to emulate functionality corresponding to each ofvirtual router devices 270 a and 270 b.

FIG. 2D illustrates a further example of managing ongoing communicationsfor the virtual computer network described with respect to FIGS. 2A-2C,but with communications being managed to support virtual networkingfunctionality for the managed virtual computer network in accordancewith a configured network topology for the virtual computer network thatincludes one or more computing nodes of the virtual computer networkthat act as intermediate nodes that facilitate handling at least someinter-node communications. In particular, FIG. 2D illustrates computingnodes A and H, Communication Manager modules R and S, System Managermodule 290, and interconnection network 250 in a manner similar to thatshown in FIGS. 2A-2C. However, in the example of FIG. 2D, the virtualcomputer network is configured to have at least two logical subnets, andcomputing node H is configured to act as an intermediate computing nodethrough which inter-node communications between the two logical subsetsare passed (e.g., as a firewall device between the two logical subnets).One or more virtual networking devices may further be configured as partof the network topology, such as illustrated with respect to FIG. 2C,but are not illustrated or explicitly managed in the example of FIG. 2D.

In the example of FIG. 2D, computing node A is sending a communicationto computing node C, with computing nodes A and C being configured tobelong to distinct logical subnets of the virtual computer network, andwith computing node H being configured to act as a firewall device forcommunications passing between those logical subnets. The actions of thephysically implemented modules 210 and 260 and devices of network 250 inactually sending the communication are shown, as well as the actions ofcomputing node H in facilitating the sending of the communication. Whileonly a single intermediate computing node is illustrated in FIG. 2D asbeing used in the routing path of the virtual computer network forinter-node communications between the two logical subnets, it will beappreciated that such a routing path may instead include 0 suchintermediate nodes (e.g., as discussed with respect to FIGS. 2A and 2B)or more than 1 such intermediate nodes in other situations, and thatsuch intermediate nodes may perform other types of actions in somesituations.

Thus, in a manner similar to that described with respect to FIG. 2A,computing node A determines to send a communication to computing node C,and accordingly exchanges various messages 224 with CommunicationManager module R 210. In particular, in this example, computing node Afirst sends a message request 224-a for virtual hardware addressinformation for computing node C. However, unlike the example of FIG. 2Ain which computing nodes A and G were part of the same logical subnetand did not include any intermediate computing nodes in the routing pathbetween computing nodes A and G for the virtual computer network,communications from computing node A to computing node C are configuredto first pass through an initial intermediate computing node H beforebeing forwarded to computing node G. Accordingly, the informationmaintained by the System Manager module 290 for use in directingcommunications from computing node A to computing node C reflects thatcomputing node H is an initial intermediate destination for suchcommunications.

Thus, Communication Manager module R intercepts the request 224-a, andobtains a hardware address to provide to computing node A as part ofspoofed ARP response message 224-b for use in directing the inter-nodecommunication along the configured routing path, in a manner similar tothat previously discussed. The Communication Manager module R may storevarious hardware address information as part of mapping information 212d, and if so may already have stored hardware address information foruse with communications from computing node A to computing node C. Ifnot, however, Communication Manager module R performs one or moreinteractions 229 with the System Manager module 290 to obtaininformation from the module 290 corresponding to the indicated virtualnetwork address for computing node C. Rather than providing thesubstrate network address corresponding to computing node C, however, asperformed for computing node G in FIG. 2A, the System Manager module 290instead indicates that the virtual network address for computing node C(at least for communications sent from computing node A) corresponds tothe substrate network address for computing node H as part of thesubstrate network routing path to computing node C, and may also provideinformation to the Communication Manager module R that indicates ahardware address to use to represent computing node C (at least forcommunications sent from computing node A). In particular, as discussedwith respect to FIG. 2C, the System Manager module 290 maintains variousinformation 296 related to the configured network topology for thevirtual computer networks that it provides or otherwise manages (e.g.,as generated at least in part by a separate Network Routing Managermodule, not shown, as described in greater detail with respect to FIG.2E and elsewhere), such as information about intermediate computingnodes along routing paths between particular computing nodes, and usesthat information to provide requested information to CommunicationManager modules. The Communication Manager module R then stores thereceived information as part of mapping information 212 d for futureuse, with the hardware address for computing node C being associatedwith the substrate network address for computing node H (at least forcommunications sent from computing node A), and provides computing nodeA with the hardware address corresponding to computing node C as part ofresponse message 224-b.

After receiving the response message 224-b from Communication Managermodule R, computing node A creates and initiates the sending of acommunication to computing node C, shown in FIG. 2D as communication224-c. In particular, the header of communication 224-c includes adestination network address for destination computing node C that is“10.0.5.1”, a destination hardware address that is the virtual hardwareaddress for computing node C provided to computing node A in message224-b, a source network address for sending computing node A that is“10.0.0.2”, and a source hardware address for sending computing node Athat is an actual or dummy hardware address that was previouslyidentified to computing node A.

The outgoing communication 224-c is intercepted and handled byCommunication Manager module R in a manner similar to that previouslydescribed with respect to FIGS. 2A and 2C. In particular, as withcommunication 220-c in FIG. 2A, Communication Manager module Rintercepts the communication 224-c, modifies the communication asappropriate, and forwards the modified communication over theinterconnection network 250 to computing node H. To determine thesubstrate network address to be used for forwarding the modifiedcommunication over the interconnection network 250, CommunicationManager module R extracts the destination virtual network address anddestination virtual hardware address from the header of communication224-c. After Communication Manager module R determines the IPv6 actualphysical substrate network address corresponding to computing node H, itcreates a new IPv6 header that includes that actual physical substratenetwork address as the destination address, and similarly adds a sourceIPv6 address for computing node A to the new header. The CommunicationManager module R next creates a new communication 234-3 by modifyingcommunication 224-c so as to replace the prior IPv4 header with the newIPv6 header (e.g., in accordance with SIIT), including populating thenew IPv6 header with other information as appropriate for the newcommunication (e.g., payload length, traffic class packet priority,etc.), and forwards communication 234-3 over the interconnection network250. As discussed in greater detail with respect to FIG. 2B, in responseto the request message 224-a and/or communication 224-c, theCommunication Manager module R and/or the System Manager module 290 mayfurther perform various optional authentication activities.

The interconnection network then uses the physical IPv6 destinationnetwork address of the communication 234-3 to route the communication toCommunication Manager module S. When Communication Manager module Sreceives communication 234-3 via the interconnection network 250, itperforms actions similar to those described in FIG. 2A with respect tocommunication 230-3, including to optionally perform interactions 244with the System Manager module 290 to determine if the communication isauthorized, to update mapping information 262 d to reflect any newinformation about computing node A, to modify the communication toinclude an appropriate IPv4 header, and to provide the modifiedcommunication as communication 248-e to computing node H. However, thecommunication 248-e provided to computing node H includes informationthat indicates that computing node C is the final destination for thecommunication, such as for use by computing node H in performing itsfirewall analysis. In particular, in this example, the communication248-e includes a destination hardware address that corresponds tocomputing node H, but the destination network address is the virtualnetwork address for computing node C, in a manner similar tocommunication 222-c/265 a of FIG. 2C.

When computing node H receives communication 248-e, it optionallyperforms various firewall-related activities for the communication,based on its configuration, and in this example determines to forwardthe communication on to its final destination of computing node C. Asdiscussed in greater detail elsewhere, such intermediate computing nodesvia which some inter-node communications may be directed may provide avariety of other types of capabilities in other embodiments andsituations. Furthermore, as noted elsewhere, computing node H may insome such situations determine to modify the communication in one ormore manners based on its firewall policies. In order to forward thecommunication on to computing node C, computing node H updates thereceived communication 248-e so that it has a new destination hardwareaddress that corresponds to computing node C (optionally afterperforming interactions with Communication Manager module S to obtainthe hardware address for computing node C's virtual network address, notshown, in a manner similar to that of communications 224-a and 224-b).The computing node H then sends the modified communication as outgoingcommunication 248-f.

In a manner similar to that previously discussed elsewhere, the outgoingcommunication 248-f is intercepted and handled by Communication Managermodule S. In particular, Communication Manager module S intercepts thecommunication 248-f, modifies the communication as appropriate, andforwards the modified communication over the interconnection network 250to computing node C. To determine the substrate network address to beused for forwarding the modified communication over the interconnectionnetwork 250, Communication Manager module S extracts the destinationvirtual network address and destination virtual hardware address fromthe header of communication 248-f. After Communication Manager module Sdetermines the IPv6 actual physical substrate network addresscorresponding to computing node C, it creates a new IPv6 header thatincludes that actual physical substrate network address as thedestination address, and similarly adds a source IPv6 address forcomputing node A to the new header. The Communication Manager module Snext creates a new communication 234-6 by modifying communication 248-fso as to replace the prior IPv4 header with the new IPv6 header (e.g.,in accordance with SIIT), including populating the new IPv6 header withother information as appropriate for the new communication (e.g.,payload length, traffic class packet priority, etc.), and forwardscommunication 234-6 over the interconnection network 250.

The interconnection network then uses the physical IPv6 destinationnetwork address of the communication 234-6 to route the communication toCommunication Manager module R. When Communication Manager module Rreceives communication 234-6 via the interconnection network 250, itperforms actions similar to those previously described, including tooptionally determine if the communication is authorized, to modify thecommunication to include an appropriate IPv4 header, and to provide themodified communication as communication 224-h to computing node C. Asdiscussed in greater detail elsewhere, in other embodiments thesubstrate network that is used may instead use the same networkaddressing scheme as the overlay virtual computer network, such as toboth use IPv4.

In this manner, the CNS system may provide virtual networkingfunctionality corresponding to the configured network topology,including to support intermediate computing nodes along routing pathsbetween particular computing nodes, and again without any specialconfiguration of the computing nodes of the managed virtual computernetwork or of the physical networking devices of the interveningsubstrate interconnection network, based on the Communication Managermodules overlaying the virtual computer network on the actual physicalinterconnection network in such a manner as to emulate the configurednetwork topology.

FIG. 2E continues the example of FIG. 2D, in which computing node Hacted as an intermediate computing node for at least some inter-nodecommunications of a managed virtual computer network by facilitatinghandling of those communications, and in particular performedfirewall-related functionality for such communications between twological configured subnets as part of the example of FIG. 2D. In theexample of FIG. 2E, computing node H continues to act as such anintermediate computing node for at least some inter-node communications,and is further configured to participate in one or more defined routingprotocols for the managed virtual computer network, such as to exchangerouting-related communications with other specified networking devicesof the managed virtual computer network that include routing informationfor the managed virtual computer network. Such other specifiednetworking devices of the managed virtual computer network may include,for example, virtual router devices 270 a and 270 b, as discussed ingreater detail with respect to FIG. 2C. In addition, the types ofrouting information that computing node H may send to othercommunications may include, for example, to announce on a first virtualnetwork interface for a first of the two logical subnets that thecomputing nodes of the second other subnet are reachable or otherwiseavailable via computing node H on that first virtual network interface,and to similarly announce on a second virtual network interface for asecond of the two logical subnets that the computing nodes of the otherfirst subnet are reachable or otherwise available via computing node Hon that second virtual network interface.

Accordingly, in the example of FIG. 2E, computing node H has determinedto send a routing-related communication to virtual router L 270 b (e.g.,based on virtual router L being configured in the network topology forthe managed virtual computer network as being a nearby neighbor routerthat is part of one of the two logical subnets, or instead in othermanners), such as to include various routing information for the managedvirtual computer network that is locally stored by or otherwiseaccessible to computing node H. Thus, in this example, computing node Hcreates the routing-related communication, and initiates sending of therouting-related communication 269-j, which computing node H intends tosend directly to virtual router L as shown with respect to logicalcommunication 266. In particular, the routing-related communication hasa destination network address and destination hardware address that arethe virtual network address and virtual hardware address assigned tovirtual router L (in this example, “10.1.5.1” and “00-01-42-CD-11-01,”respectively), and source network address and source hardware addresscorresponding to computing node H. While not illustrated here, in orderto obtain the destination network address and/or the destinationhardware address for virtual router L, computing node H may perform oneor more other prior communications (not shown) to obtain suchinformation, such as via an ARP message request.

In a manner similar to that described previously, Communication Managermodule S intercepts the routing-related communication 269-j, anddetermines that the routing-related communication is intended for avirtual networking device that is part of the configured networktopology of the managed virtual computer network, such as based on theindicated destination virtual network address and/or destinationhardware address of the routing-related communication, or instead inother manners (e.g., based at least in part on the routing protocol usedfor the communication). As previously discussed, the CommunicationManager module S may store various hardware address information andvirtual network address information as part of mapping information 262e, and if so may already have stored information for use withcommunications directed to virtual router L. If not, however,Communication Manager module S may perform one or more interactions (notshown) with the System Manager module 290 to obtain information from themodule 290 corresponding to the virtual router L, in a manner similar tothat discussed previously.

However, in a manner different from that of FIGS. 2A-2D, rather thanforwarding the received communication 269-j over the substrate networkto its intended destination, Communication Manager module S determinesto provide the received routing-related communication 269-j (or routinginformation from the communication) to the Network Routing Managermodule 295, such as via one or more interactions 243 e. While the module295 is not explicitly illustrated as being physically connected to theinterconnection network 250 in this example, such physical connectionsmay exist in at least some embodiments, in a manner similar to thatpreviously discussed for System Manager module 290 and the variousCommunication Manager modules. Accordingly, while the interactions 243 ewith the module 295 are not illustrated in this example as explicitlytraveling over the interconnection network 250, it will be appreciatedthat such interactions may include one or more communications that dotravel over the interconnection network 250 in at least someembodiments. In addition, as discussed in greater detail elsewhere, insome embodiments, the Communication Manager module S and/or the NetworkRouting Manager module 295 may filter or restrict at least some types ofrouting-related communications, such as by discarding or otherwise notusing such communications, and thus the Communication Manager module Sand/or the Network Routing Manager module 295 may further performvarious optional activities in determining whether to use the routinginformation in the received communication 269-j.

As discussed in greater detail elsewhere, the Network Routing Managermodule 295 may use the routing information from the routing-relatedcommunication in various manners, including to determine whether therouting information indicates any changes in the configured networktopology for the managed virtual computer network. As previously noted,the System Manager module 290 maintains various information 296 relatedto the configured network topology for the virtual computer networksthat it provides or otherwise manages (e.g., as generated at least inpart by the Network Routing Manager module 295), such as informationabout intermediate computing nodes along routing paths betweenparticular computing nodes and information about specified targetnetwork addresses and associated computing nodes, and uses thatinformation to provide requested information to Communication Managermodules. Accordingly, the Network Routing Manager module 295 may analyzethe received routing information to determine if it reflects anyrelevant changes to the configured network topology information for themanaged virtual computer network (e.g., any changes regardingintermediate computing nodes along particular inter-node communicationrouting paths, any changes regarding the use of particular targetnetwork addresses by particular computing nodes, any changes regardingparticular public network addresses that are externally associated withmanaged computer network, etc.). If the Network Routing Manager module295 identifies any such relevant changes to the configured networktopology information for the managed virtual computer network, theNetwork Routing Manager module 295 performs one or more interactions 297e (optionally by sending one or more communications, not shown, over theinterconnection network 250) with the System Manager module 290 to causethe network topology information 296 to be updated in a correspondingmanner. In addition, if changes regarding one or more particular publicnetwork addresses that are externally associated with the managedcomputer network have been indicated in the routing information, theNetwork Routing Manager module 295 may take further actions that are notshown, such as to initiate a public announcement to external computersystems of those public network addresses.

In a similar manner, after the managed virtual computer network is inthe use, the entity Z client 285 associated with the managed virtualcomputer network may optionally in some embodiments perform additionalinteractions with the System Manager module 290 to provide updates orother new configured network topology information for the managedvirtual computer network, or in some embodiments to configureinformation related to the use of particular target virtual computernetwork addresses for the managed virtual computer network. If so, theSystem Manager module 290 may store updated information corresponding tothe indicated changes and/or may provide the new network topologyinformation and/or other information to the Network Routing Managermodule 295 (e.g., via one or more interactions 297 e) for analysis. Ifprovided to the Network Routing Manager module 295, the module 295 mayproceed to perform actions similar to those performed for the receivedrouting information from routing-related communication 269-j, includingto analyze the received new information to determine if it reflects anyrelevant changes to the configured network topology information for themanaged virtual computer network. If the Network Routing Manager module295 identifies any such relevant changes to the configured networktopology information for the managed virtual computer network, theNetwork Routing Manager module 295 similarly performs one or moreinteractions 297 e with the System Manager module 290 to cause thenetwork topology information 296 to be updated in a correspondingmanner.

In addition to updating the stored network topology information 296 asappropriate, the Network Routing Manager module 295 may further takeother actions in some embodiments. For example, if changes to thenetwork topology are detected that correspond to changes to one or moreintermediate computing nodes for use in routing paths for inter-nodecommunications between at least some computing nodes, the NetworkRouting Manager module 295 may optionally perform interactions 246 tosupply corresponding update information to one or more CommunicationManager modules that manage computing nodes affected by the networktopology changes. If such updated information is supplied toCommunication Manager modules R and/or S, for example, those modules mayuse that updated information to update their stored mapping information212 e and 262 e, respectively, in a corresponding manner for use withfuture inter-node communications that are affected by the networktopology change. In other embodiments, the Communication Manager modulesmay instead obtain such updated network topology information directlyfrom the System Manager module 290, such as if the Communication Managermodules periodically contact the System Manager module 290 for currentinformation (e.g., if the stored mapping information 212 e and 262 eexpires periodically or instead based on other criteria, such as basedon a communication from the System Manager module 290 and/or the NetworkRouting Manager module 295 that some or all of the stored mappinginformation is no longer valid).

In addition, if the routing information in routing communication 269-jrelates to a target virtual network address that is configured for useas an anycast address, the Network Routing Manager module 295 mayperform interactions 297 e and 246 to update information for the managedcomputer network in a corresponding manner. For example, if computingnode H has newly announced in routing communication 269-j that theanycast target virtual network address is available via computing nodeH, such as if computing node H is configured to operate as one of thegroup of alternative computing nodes associated with that anycastaddress, the Network Routing Manager module 295 updates the networktopology information 296 to reflect that current association, and mayfurther perform operations to determine whether to use computing node Hfor some or all communications directed to the anycast target virtualnetwork address for the managed computer network, as discussedelsewhere. If the module 295 determines that computing node H is to beused for communications directed to the anycast target virtual networkaddress from one or more of the computing nodes of the managed computernetwork, the interactions 246 may include updating the CommunicationManager modules that are associated with those one or more computingnodes of the new association, so that future communications from thosecomputing nodes for that anycast target virtual network address will bedirected to computing node H. In addition, in some situations, themodule 295 may determine that computing node H has withdrawn from beingone of the group of computing nodes associated with the anycast targetvirtual network address, such as based on not having received anannouncement from computing node H that advertises the anycast targetvirtual network address for a defined period of time, or by otherwisereceiving an explicit indication of withdrawal of computing node H fromthe anycast group of computing nodes.

Furthermore, if the routing information in routing communication 269-jrelates to a target virtual network address that is available forcomputing node H to take ownership of (e.g., a particular virtualnetwork address for which computing node H has been designated as apossible owner; any virtual network address and any computing node, suchas based on a corresponding prior configuration of the managed computernetwork; etc.), the Network Routing Manager module 295 may performinteractions 297 e and 246 to update information for the managedcomputer network in a corresponding manner. For example, if computingnode H has newly announced in routing communication 269-j that thetarget virtual network address is available via computing node H, suchas if computing node H is taking over ownership of that target virtualnetwork address from another computing node that previously used thattarget virtual network address, the Network Routing Manager module 295may optionally verify that computing node H is authorized to performthat action. If the module 295 determines that computing node H will beassociated with the target virtual network address for communicationsthat are directed to that target virtual network address from or morecomputing nodes of the managed computer network, the module 295 updatesthe network topology information 296 to reflect that currentassociation, and may perform interactions 246 that include updating theCommunication Manager modules that are associated with those one or morecomputing nodes of the new association, so that future communicationsfrom those computing nodes for that target virtual network address willbe directed to computing node H. As discussed in greater detailelsewhere, such routing communications from computing node H may bespecified in various manners, including in some embodiments andsituations by using GARP messages. In addition, in some situations, themodule 295 may determine that computing node H has withdrawn from beingthe computing node that was associated with a particular target virtualnetwork address, such as based on not having received an announcementfrom computing node H that advertises the target virtual network addressfor a defined period of time, or by otherwise receiving an explicitindication of withdrawal of computing node H from use of that targetvirtual network address.

Moreover, if the routing information in routing communication 269-jrelates to one or more public network addresses that are available forexternal use with computing node H or otherwise with the managedcomputer network, the Network Routing Manager module 295 may performinteractions to update information for the managed computer network in acorresponding manner. For example, if computing node H has newlyannounced one or more such public network addresses in routingcommunication 269-j, the Network Routing Manager module 295 mayoptionally verify that the indicated public network addresses areauthorized to be used, as discussed in greater detail elsewhere. Suchpublic network addresses may be indicated in various manners, includingusing the BGP routing protocol. If the module 295 determines that use ofthe indicated public network addresses are authorized, the module 295updates the network topology information 296 to reflect thatinformation, and may perform interactions 246 not shown to externallyannounce those indicated public network addresses, such as byinteracting with remote capabilities provided by the client entity Zthat manage such public network addresses, by interacting with an edgeCommunication Manager module (not shown) that manages communicationsbetween a geographical location (e.g., a data center) at which themanaged computer network is located and other external computer systemsat other geographical locations, etc.

Furthermore, while not illustrated in FIG. 2E, the Network RoutingManager module 295 may take other actions in at least some embodimentsand situations, such as to initiate the sending of routing-relatedcommunications to computing nodes such as computing node H thatparticipate in the predefined routing protocol for the managed virtualcomputer network. Such sending of a routing-related communication tocomputing node H or other similar computing node may be initiated by theNetwork Routing Manager module 295 in various manners in variousembodiments, such as by generating a particular routing-relatedcommunication and forwarding it over the interconnection network 250 tothe destination computing node in a manner similar to that of otherinter-node communications (optionally with the sent communicationincluding sender information corresponding to a virtual networkingdevice for the managed virtual computer network, such as virtual routerL or virtual router J, and with the included routing information beingprovided in a manner that is contextualized to be appropriate for thatindicated sender). Alternatively, the Network Routing Manager module 295may in some embodiments and situations instead provide the routinginformation intended for such a computing node to the computing node'sassociated Communication Manager module (e.g., to Communication Managermodule S for computing node H), and have that associated CommunicationManager module generate and provide an appropriate routing-relatedcommunication with that routing information to the associated computingnode. As discussed in greater detail elsewhere, such routing informationprovided to computing nodes may have various forms, including thefollowing: routing information received from other computing nodes orother sources (e.g., a remote networking device at a client's remotenetwork that interacts with the managed virtual computer network) thatis being forwarded to additional computing nodes; routing informationthat reflects changes made by the client to the configured networktopology for the managed virtual computer network, such as viainteractions with the System Manager module; routing information that isgenerated by the Network Routing Manager module to provide globallyconsistent routing information for all computing nodes of the managedvirtual computer network, or that otherwise satisfies a centralized goalfor operation of the managed virtual computer network; etc.

In this manner, the CNS system may further provide virtual networkingfunctionality corresponding to the configured network topology,including to support intermediate computing nodes along routing pathsbetween particular computing nodes, various types of use of targetvirtual network addresses, and use of indicated public networkaddresses, and again without any special configuration of the computingnodes of the managed virtual computer network or of the physicalnetworking devices of the intervening substrate interconnection network.Such techniques may be based at least in part on the CommunicationManager modules managing at least some routing-related communications(e.g., routing-related communications directed to virtual networkingdevices) in appropriate manners, and/or on a Network Routing Managermodule using configured network topology information and/or dynamicallysupplied routing information to manage inter-node communications for amanaged virtual computer network.

Furthermore, as previously noted, the configurable network service mayin some embodiments operate as a fee-based service, such that clientsare customers that pay fees to the configurable network service. In suchembodiments, the configurable network service may charge clients usingvarious metrics and for various types of functionality that is provided.For example, metrics on which fees may be based in at least someembodiments include, for example, one or more of the following: based ona size of routing communication traffic sent and/or received via virtualnetworking devices; based on a quantity or rate of routing changes orother routing communications that are sent to or via a virtualnetworking device; based on the types of communications that are sentvia a virtual networking device, including for various types ofannouncements or specialized types of communications (e.g., anycastcommunications, broadcast communications, etc.); for the configurablenetwork service to enforce client-specified routing filters or otherspecified restrictions on types of communications that are allowed ordisallowed via a virtual networking device; for the configuration,dynamic indication to use and/or ongoing actual use of anycast targetvirtual network addresses; for the configuration, dynamic indication touse and/or ongoing actual use of target virtual network addresses thatsome or all computing nodes may claim ownership of; for the dynamicindication to use and/or ongoing use of external public networkaddresses; for a virtual peering router provided by the configurablenetwork service to provide functionality to manage inter-connectionsbetween two or more managed computer networks; etc. Furthermore, feesthat are charged may have various forms, including one-time fees,ongoing subscription fees (e.g., with a certain fee for a specifiedperiod of time), usage-based fees (e.g., to vary with an amount of useof a particular type), etc.

In addition, as previously noted, various computing nodes may beselected for a managed virtual computer network and configured inaccordance with a configured network topology of that managed virtualcomputer network in various manners. For example, in some embodiments,the selection of a computing node to be used in a managed virtualcomputer network and/or to be assigned a particular role in a configurednetwork topology may be based at least in part on a geographical and/ornetwork location of the computing node, such as an absolute location, orinstead a location relative to one or more other computing resources ofinterest (e.g., other computing nodes of the same managed virtualcomputer network, storage resources to be used by the computing node,etc.), such as within a minimum and/or maximum specified geographicaldistance or other degree of proximity to an indicated other computingresource or other location. In addition, in some embodiments, factorsused when selecting a computing node may be based in part or in whole onfactors other than location, such as to include one or more of thefollowing: constraints related to capabilities of a computing node, suchas resource-related criteria (e.g., an amount of memory, an amount ofprocessor usage, an amount of network bandwidth, and/or an amount ofdisk space), and/or specialized capabilities available only on a subsetof available computing nodes; constraints related to costs, such asbased on fees or operating costs associated with use of particularcomputing nodes; etc.

As previously noted, in some embodiments, a program execution serviceexecutes third-party customers' programs using multiple physicalcomputing systems (e.g., in one or more data centers) that each hostmultiple virtual machines, with each virtual machine being able toexecute one or more programs for a customer. In some such embodiments,customers may provide programs to be executed to the program executionservice, and may reserve execution time and other resources on physicalor virtual hardware facilities provided by the program executionservice. In addition, customers and/or the program execution service maydefine virtual computer networks that will be used by the programexecution service for computing nodes of the customer, so as totransparently provide computing nodes of a virtual computer network withthe appearance of operating on a dedicated physical network.

In addition, in some situations, a communication manager module maytrack or otherwise determine the virtual computer networks to which themodule's associated computing nodes belong as part of managing thecommunications for the virtual computer networks. The determination by acommunication manager module of a corresponding virtual computer networkfor a computing node may be performed in various ways in variousembodiments, such as by interacting with a system manager module thatprovides that information, by tracking software programs executing onsuch computing nodes, by tracking entities associated with suchcomputing nodes, etc. For example, when a particular computing nodebegins to execute one or more software programs on behalf of a user, andthat user also has other software programs executing on other computingnodes, the new computing node executing the user's program(s) may beselected to be associated with a virtual computer network for the userthat includes those other computing nodes. Alternatively, a user orother entity may specify a particular managed computer network to whicha computing node belongs, such as if the entity maintains multipledistinct managed computer networks between different groups of computingnodes.

Various other types of actions than those discussed with respect toFIGS. 2A-2E may be performed in other embodiments, including for typesof network addressing protocols other than IPv4 and IPv6.

FIG. 3 is a block diagram illustrating example computing systemssuitable for executing an embodiment of a system for managingcommunications between computing nodes. In particular, FIG. 3illustrates a group 399 of computing systems and inter-network(s), suchas a data center or other group of co-located computing nodes. In someembodiments, some or all of the computing systems of the group 399 maybe used by an embodiment of the CNS system to provide managed virtualcomputer networks to users or other entities. The group 399 includes aserver computing system 300, a host computing system 350 capable ofexecuting one or more virtual machines, other host computing systems 390that are similar to host computing system 350, and an optionalCommunication Manager module 360 that manages host computing systems 390and that executes on one of the computing systems 390 or on anothercomputing system (not shown). The server computing system 300 and hostcomputing systems 350 and 390 are connected to one another via aninternal network 380, which includes a physical networking device 362and other networking devices (not shown). The network 380 may be aninterconnection network that joins multiple disparate physical networks(not shown) for the group 399 and possibly provides access to externalnetworks (not shown) and/or systems, such as other computing systems395. In the illustrated example, the networking device 362 provides agateway from the network 380 to the host computing systems 350 and 390.In some embodiments, networking device 362 may, for example, be a routeror a bridge.

The computing system 300 operates to configure and manage virtualcomputer networks within the group 399, as well as to provide otherfunctions (e.g., the provisioning, initialization, and execution ofprograms on computing nodes), including the managing of functionalitycorresponding to use of received routing information. The computingsystem 300 includes a CPU 305, various I/O components 310, storage 330,and memory 320. The I/O components include a display 311, networkconnection 312, computer-readable media drive 313, and other I/O devices315 (e.g., a mouse, keyboard, speakers, etc.).

The host computing system 350 operates to host one or more virtualmachines, such as for use as computing nodes in managed virtual computernetworks (e.g., computing nodes that execute programs on behalf ofvarious users). The host computing system 350 includes a CPU 352,various I/O components 353, storage 351, and memory 355. While notillustrated here, the I/O components 353 may include similar componentsto those of I/O components 310. A virtual machine Communication Managermodule 356 and one or more virtual machines 358 are executing in thememory 355 (and may each have an associated amount of storage 351 and/orother local computing resources), with the module 356 managingcommunications for the associated virtual machine computing nodes 358.The Communication Manager module 356 maintains various mappinginformation 354 on storage related to the computing nodes 358 and othercomputing nodes, such as in a manner similar to mapping information 212and 262 of FIGS. 2A-2E. The structure of the other host computingsystems 390 may be similar to that of host computing system 350, orinstead some or all of the host computing systems 350 and 390 may actdirectly as computing nodes by executing programs without using hostedvirtual machines. In a typical arrangement, the group 399 may includehundreds or thousands of host computing systems such as thoseillustrated here, organized into a large number of distinct physicalsub-networks and/or networks.

An embodiment of a CNS system 340 is executing in memory 320 of thecomputing system 300. In some embodiments, the system 340 may receive anindication of multiple computing nodes to be used as part of a managedvirtual computer network (e.g., one or more virtual machine computingnodes on host computing system 350 or one or more computing nodes usingone of the host computing systems 390), and in some situations mayselect the particular computing node(s) for the managed virtual computernetwork. In some cases, information about the structure and/ormembership of various managed virtual computer networks may be stored inthe provisioning database 332 on storage 330 by the system 340, such asin a manner similar to information 292 of FIGS. 2A-2D, and provided tothe Communication Manager modules at various times. Similarly, in somecases, information about the configured network topology of variousmanaged computer networks (e.g., information about logical subnets,virtual networking devices, intermediate computing nodes along routingpaths for communications between particular computing nodes, connectionsto other managed computer networks via one or more virtual peeringrouters, configured target virtual network addresses and associatedcomputing nodes, etc.) may be stored in the database 334 on storage 330by the system 340, such as in a manner similar to information 296 ofFIGS. 2C-2E, and provided to the Communication Manager modules atvarious times. In this example, the system 340 in memory 320 includes aNetwork Routing Manager (“NRM”) module 342 and optionally other modules344 (e.g., a system manager module, a payment module to obtain fees fromclients if the system manager module does not include suchpayment-related functionality, etc.), with the communication managermodules 356 and 360 being a further part of the distributed CNS system.The NRM module 342 performs operations to facilitate the configurationand use of specified network topology for managed computer networks,such as in response to requests from clients and/or dynamically receivedrouting information, as discussed elsewhere. In at least someembodiments, the NRM module 342, other modules 344, and/or variouscommunication manager modules may each include software instructionsthat may be stored on long-term non-volatile storage (e.g., storage 330,storage 351, etc.) and may be loaded into memory (e.g., memory 320,memory 355, etc.) for execution by at least one of one or moreprocessors (e.g., CPU 305, CPU 352, etc.), so as to program or otherwiseconfigure those processors and the corresponding computingsystems/devices to perform the described techniques.

As discussed in greater detail elsewhere, the Communication Managermodules 356 and 360 (and other Communication Manager modules, not shown,that manage other associated computing nodes, not shown) and the variousmodules 342 and 344 of the system 340 may interact in various ways tomanage communications between computing nodes, including to providevirtual networking functionality corresponding to configured networktopologies for provided virtual computer networks, including withrespect to target virtual network addresses and associated computingnodes. Such interactions may, for example, enable the computing nodes358 and/or other computing nodes to inter-communicate over managedvirtual computer networks without any special configuration of thecomputing nodes, by overlaying the virtual computer networks overnetwork 380 and optionally one or more external networks (not shown)without any special configuration of networking device 362 or othernetworking devices (not shown), and optionally without encapsulation ofcommunications.

It will be appreciated that computing systems 300, 350, 390, and 395,and networking device 362, are merely illustrative and are not intendedto limit the scope of the present invention. For example, computingsystems 300 and/or 350 may be connected to other devices that are notillustrated, including through one or more networks external to thegroup 399, such as the Internet or via the World Wide Web (“Web”). Moregenerally, a computing node or other computing system may comprise anycombination of hardware or software that can interact and perform thedescribed types of functionality, including without limitation desktopor other computers, database servers, network storage devices and othernetwork devices, PDAs, cell phones, wireless phones, pagers, electronicorganizers, Internet appliances, television-based systems (e.g., usingset-top boxes and/or personal/digital video recorders), and variousother consumer products that include appropriate communicationcapabilities. In addition, the functionality provided by the illustratedmodules may in some embodiments be combined in fewer modules ordistributed in additional modules, such as if the functionality of asystem manager module and a network routing manager module are insteadcombined into a single module. Similarly, in some embodiments thefunctionality of some of the illustrated modules may not be providedand/or other additional functionality may be available. Furthermore, insome embodiments the functionality of a single illustrated module may bedistributed across multiple related modules that provide the describedfunctionality in an aggregate manner.

It will also be appreciated that, while various items are illustrated asbeing stored in memory or on storage while being used, these items orportions of them may be transferred between memory and other storagedevices for purposes of memory management and data integrity.Alternatively, in other embodiments some or all of the modules and/orsystems may execute in memory on another device and communicate with theillustrated computing systems via inter-computer communication.Furthermore, in some embodiments, some or all of the systems and/ormodules may be implemented or provided in other manners, such as atleast partially in firmware and/or hardware, including, but not limitedto, one or more application-specific integrated circuits (ASICs),standard integrated circuits, controllers (e.g., by executingappropriate instructions, and including microcontrollers and/or embeddedcontrollers), field-programmable gate arrays (FPGAs), complexprogrammable logic devices (CPLDs), etc. Some or all of the modules,systems and data structures may also be stored (e.g., as softwareinstructions or structured data) on a computer-readable medium, such asa hard disk, a memory, a network, or a portable media article to be readby an appropriate drive or via an appropriate connection. The systems,modules and data structures may also be transmitted as generated datasignals (e.g., as part of a carrier wave or other analog or digitalpropagated signal) on a variety of computer-readable transmissionmediums, including wireless-based and wired/cable-based mediums, and maytake a variety of forms (e.g., as part of a single or multiplexed analogsignal, or as multiple discrete digital packets or frames). Suchcomputer program products may also take other forms in otherembodiments. Accordingly, the present invention may be practiced withother computer system configurations.

FIG. 4 is a flowchart of an example embodiment of a CNS System Managerroutine 400. The routine may be provided by, for example, execution ofthe system manager module 110 of FIG. 1B, the manager module 110 of FIG.1A, the system manager module 290 of FIGS. 2A-2E, and/or a systemmanager module (not shown) of CNS system 340 of FIG. 3, such as toassist in managing communications between multiple computing nodesacross one or more intermediate networks, including to managecommunications so as to provide virtual networking functionalitycorresponding to configured network topologies of managed computernetworks, including based on use of target virtual network addresses andassociated computing nodes, as well as to perform other types ofmanagement operations in some situations. In at least some embodiments,the routine may be provided as part of a system that managescommunications for multiple different entities across a commonintermediate network, with the communications configured so as to enableeach computing node to transparently communicate with other associatedcomputing nodes using a private virtual computer network that isspecific to that entity. Furthermore, the routine may facilitatepreventing unauthorized communications from being provided todestination computing nodes, such as by assisting Communication Managermodules with determinations of whether communications are authorized.

In the illustrated embodiment, the routine begins at block 405, where arequest is received. The routine continues to block 410 to determine thetype of request. If it is determined that the type of request is toassociate one or more computing nodes with a particular managed virtualcomputer network provided for an indicated entity, such as if thosecomputing nodes are executing or are to execute one or more programs onbehalf of that entity, the routine continues to block 415 to associatethose computing nodes with that indicated entity and virtual computernetwork. In some embodiments, the routine may further determine the oneor more computing nodes to be associated with the indicated entity andvirtual computer network, such as based on information provided by theindicated entity, while in other embodiments the selection of suchcomputing nodes and/or execution of appropriate programs on thosecomputing nodes may be performed in other ways. In addition, asdiscussed in greater detail elsewhere, in some embodiments one or moreof the computing nodes may each be a virtual machine that is hosted byone or more physical computing systems. The routine then continues toblock 420 to store mapping information for the computing nodes and themanaged virtual computer network. In particular, in the illustratedembodiment the routine stores for each computing node an indication of aphysical substrate network address corresponding to the computing node,a virtual network address used by the entity for the computing node aspart of the virtual computer network, optionally a virtual hardwareaddress assigned to the computing node, and an indication of theassociated entity. As discussed in greater detail elsewhere, thephysical substrate network address corresponding to the computing nodemay in some embodiments be a substrate network address specific to thatsingle computing node, while in other embodiments may instead refer to asub-network or other group of multiple computing nodes, such as may bemanaged by an associated Communication Manager module. In addition, thespecification of various network topology information for computingnodes and virtual networking devices is discussed in greater detail withrespect to block 480 and FIGS. 6A-6B (including to use target virtualnetwork addresses), such as to be further included in the stored mappinginformation as discussed in greater detail previously, although in otherembodiments such information may further be received and stored withrespect to block 420. After block 420, the routine continues to block422 to optionally provide information about the computing nodes andtheir configuration to one or more communication manager modulesassociated with those computing nodes, although in other embodimentsinstead provides such information upon request from the communicationmanager modules.

If it is instead determined in block 410 that the type of receivedrequest is a request for address resolution for a virtual networkaddress of a target computing node or other network device of interest,such as from a communication manager module on behalf of a managedcomputing node, the routine continues instead to block 425, where itdetermines whether the request is authorized in one or more ways, suchas based on whether the managed computing node on whose behalf therequest is made is authorized to send communications to a computing nodewhose virtual network address resolution is requested (e.g., based onthe virtual computer network(s) to which the two computing nodes belong,and optionally any inter-connections between multiple such virtualcomputer networks via one or more virtual peering routers), based onwhether the managed computing node on whose behalf the request is madeis a valid computing node that is currently part of a configured virtualcomputer network, and/or based on whether the request is received fromthe communication manager module that actually manages the indicatedcomputing node on whose behalf the request is made. If the request isdetermined to be authorized, the routine continues to block 430, whereit obtains a virtual network address of interest for a particularvirtual computer network, such as may be included with the requestreceived in block 405, or previously stored and currently identifiablefor the target computing node of interest based on other receivedinformation. The routine then continues to block 435 to retrieve storedinformation for the computing node that is associated with the virtualnetwork address for the virtual computer network, and in particularinformation that associates that virtual network address to a physicalsubstrate network address for a network location that corresponds to thecomputing node, such as may be previously stored with respect to block420, and optionally to other information for the virtual network address(e.g., an associated virtual hardware address, an indication regardingwhether the virtual network address corresponds to a physicallyimplemented computing node with an actual substrate network address orinstead to a virtual networking device that does not have an actualsubstrate network address, information about a role or status of thedevice corresponding to the virtual network address with respect toconfigured network topology information, etc.). Furthermore, insituations in which the destination virtual network address correspondsto a configured target virtual network address that has multipleassociated computing nodes (e.g., an anycast target virtual networkaddress with a group of multiple alternative computing nodes, a targetvirtual network address whose ownership may be claimed by differentcomputing nodes at different times, etc.), the routine may in someembodiments perform various operations to select between one of themultiple associated computing nodes, although in other embodiments aparticular one of those multiple computing nodes may have already beenselected for communications from some or all of the computing nodes ofthe managed computer network. If communications from the computing nodeon whose behalf the request is made to the indicated computing node areto be routed via one or more intermediate computing nodes, theinformation that is provided may in part correspond to a first of thoseintermediate computing nodes, such as to include the actual substratenetwork address of that first intermediate computing node (whetherinstead of or in addition to the actual substrate network address forthe indicated computing node). After block 435, the routine continues to440 to provide an indication of the retrieved information to therequester. While not illustrated here, if the determination in block 425determines that the request is not authorized, the routine may insteadnot perform blocks 430-440 for that request, such as by responding withan error message to the request received in block 405 or not respondingto that received request. In addition, in other embodiments the routinemay perform one or more other tests to validate a received requestbefore responding with the requested information, such as to verify thatthe computing node that initiated the request is authorized to receivethat information.

If it is instead determined in block 410 that the received request is toconfigure information regarding a specified network topology for anindicated managed virtual computer network, such as from a userassociated with that virtual computer network (e.g., a userrepresentative of the client on whose behalf the virtual computernetwork is provided), the routine continues to block 480 to perform aNetwork Routing Manager routine to manage the configuration. One exampleembodiment of such a Network Routing Manager routine is described infurther detail with respect to FIGS. 6A-6B. In this illustratedembodiment, the Network Routing Manager routine manages the interactionwith the user to obtain the configured network topology information(e.g., via an API and/or GUI of the configurable network service), andstores relevant configured network topology information in a manner thatis accessible by the System Manager routine for later use (e.g., withrespect to block 435), while in other embodiments the System Managerroutine may instead manage those user interactions and/or that networktopology information storage (e.g., by providing received configurednetwork topology information to the Network Routing Manager routine foranalysis after it is received by the System Manager routine, and bystoring updated network topology information if it is received from theNetwork Routing Manager routine).

If it is instead determined in block 410 that the received request is ofanother type, the routine continues instead to block 490 to performanother indicated operation as appropriate. For example, in someembodiments, the routine may receive requests to update storedinformation about particular computing nodes, such as if a particularcomputing node was previously associated with a particular entity and/orvirtual computer network (or portion of a virtual computer network, suchas a specified logical subnet) but that association ends (e.g., one ormore programs being executed for that entity on that computing node areterminated, the computing node fails or otherwise becomes unavailable,an associated user or other client changes specified configurationinformation for the computing node, etc.). The routine may also performa variety of other actions related to managing a system of multiplecomputing nodes (e.g., charging fees to clients for particularfunctionality provided to those clients, obtaining payment from clientsfor such charged fees, etc.), as discussed in greater detail elsewhere,and may at times perform actions of other types, such as to performoccasional housekeeping operations to review and update storedinformation as appropriate (e.g., after predefined periods of time haveexpired). In addition, if possible validation problems are detected,such as with respect to received address resolution requests for virtualnetwork addresses, the routine may take various actions to signal anerror and/or perform other corresponding actions as appropriate.Furthermore, if the routine receives updated network topologyinformation for a managed virtual computer network, such as from theclient and/or from the Network Routing Manager module, the routine mayproceed to store that information for later use as appropriate.

After blocks 422, 440, 480 or 490, the routine continues to block 495 todetermine whether to continue, such as until an explicit indication toterminate is received. If it is determined to continue, the routinereturns to block 405, and if not continues to block 499 and ends.

FIGS. 5A-5B are a flow diagram of an example embodiment of a CNS

Communication Manager routine 500. The routine may be provided by, forexample, execution of the Communication Manager modules 109 a-109 d, 108and/or 150 of FIG. 1B, the Communication Manager modules 210 and/or 260of FIGS. 2A-2E, the Communication Manager modules 356 and/or 360 of FIG.3, and/or a communication manager module (not shown) of the CNS service105 of FIG. 1A, such as to manage communications to and from anassociated group of one or more computing nodes in order to provide aprivate virtual computer network over one or more shared intermediatenetworks, including to determine whether to authorize communications toand/or from the managed computing nodes, and to support providingvirtual networking functionality corresponding to configured networktopologies for managed virtual computer networks.

The routine begins at block 505, where an indication is received of anode communication or other message. The routine continues to block 510to determine the type of communication or other message and proceedaccordingly. If it is determined in block 510 that the message is arequest from an associated managed computing node for network addressresolution, such as an ARP request, the routine continues to block 515to identify the virtual network address of interest indicated in therequest. The routine then continues to block 520 to send a request to asystem manager module for virtual network address resolution for theindicated virtual network address for the virtual computer network thatis associated with the computing node that provided the request, such asdiscussed with respect to blocks 425-440 of FIG. 4. As discussed ingreater detail elsewhere, the routine may in some embodiments trackinformation about virtual computer networks and/or entities associatedwith each managed computing node, while in other embodiments at leastsome such information may instead be provided to the routine by thecomputing nodes, by the system manager module and/or by the networkrouting manager module, or instead the system manager module may trackand store that information without it being provided to and tracked bythe current routine. While not illustrated here, in other embodimentsand situations such address resolution requests may be handled in othermanners. For example, if a computing node being managed by a particularcommunication manager module provides an address resolution request foranother computing node that is also managed by that communicationmanager module, the routine may instead respond to the request withoutinteraction with the system manager module, such as based on locallystored information. In addition, while in the illustrated embodiment thereceived request is a request to provide a computing node's link-layerhardware address that corresponds to an indicated networking layeraddress, in other embodiments the address resolution request may haveother forms, or computing nodes may request other types of informationabout indicated computing nodes.

In the illustrated embodiment, the routine next continues to block 525to receive a response from the system manager module that includes aphysical substrate network address and/or other informationcorresponding to the identified virtual network address, and storesinformation locally that maps that physical substrate network addressand/or other information to a unique hardware address for later use bythe routine (e.g., based on a dummy virtual hardware address generatedby the routine or provided in the response), along with otherinformation about the computing node as discussed in greater detailelsewhere. The routine then provides the hardware address to therequesting computing node, which it may use as part of one or more latercommunications that it sends to the computing node with the indicatedvirtual network address. As discussed in greater detail elsewhere, thephysical substrate network address response that is provided may in someembodiments include a physical substrate network address that isspecific to the indicated computing node of interest, while in otherembodiments the physical substrate network address may correspond toanother intermediate computing node via which communications to theindicated computing node of interest will be passed for some or allsending computing nodes, or may instead correspond to a sub-network orother group of multiple computing nodes to which the indicated computingnode belongs, such as to correspond to another communication managermodule that manages those other computing nodes. If only some inter-nodecommunications to an indicated computing node of interest will be passedthrough a particular intermediate computing node, such as for only somesending computing nodes, it will be appreciated that the routine mayobtain and store multiple entries for the indicated computing node ofinterest, such as to include different information to use for differentsending computing nodes that initiate inter-node communications to theindicated computing node of interest. Furthermore, the association ofparticular computing nodes with particular virtual network addresses maychange in some situations, such as for target network addresses that aredynamically claimed by a new computing node or for new computing nodesthat are added to a group for an anycast target network address, and ifso the information that maps virtual network addresses to computingnodes may be updated at times. The routine next continues to block 530to determine if blocks 515-525 were performed as part of the handling ofan outgoing node communication, as discussed with respect to blocks540-560, and if so, continues to block 547, or otherwise continues toblock 595. While not illustrated here, in some embodiments the routinemay instead receive an error response from the system manager module(e.g., based on the requesting computing node not being authorized tocommunicate with the indicated destination computing node) or noresponse, and if so may not send any response or a corresponding errormessage to the requesting computing node.

If it is instead determined in block 510 that the type of communicationor other message is an outgoing communication from a computing nodemanaged by the routine, such to another indicated remote destinationcomputing node that is not managed by the routine and/or to an indicatedvirtual networking device, the routine continues to block 540. If theroutine is being performed for a communication manager module thatoperates as an edge module to manage connections between a group ofcomputing nodes connected by a substrate network (e.g., at a datacenter) and external computing systems, such outgoing communicationscorrespond to communications entering the group of inter-connectedcomputing nodes from external computing systems, and the routineproceeds to block 547 without performing blocks 540-545. In block 540,the routine identifies the indicated hardware address for thedestination from the communication header for the outgoingcommunication. In block 545, the routine then determines whether thatdestination hardware address is a hardware address previously mapped toa physical substrate network address corresponding to the destination,such as previously discussed with respect to block 525. If not, in someembodiments the routine continues to block 515 to perform blocks 515-525to determine such a corresponding physical network address for theoutgoing node communication, while in other embodiments such actions arenot performed (e.g., if the indicated hardware address is not a mappedaddress, the routine may cause the outgoing node communication to fail,such as with an error message back to the sending node).

If the indicated hardware address is a mapped address, or the check isnot performed, the routine continues to block 547 to determine if thereceived communication is a routing-related communication, such as acommunication intended for a virtual networking device. If so, theroutine continues to block 581, and otherwise continues to block 550 toretrieve the physical substrate network address that is mapped to thehardware address for the destination computing node to which thecommunication is directed (or to the destination network address forincoming communications from external computing systems). As discussedin greater detail elsewhere, if the communication is to be directed toone or more intermediate computing nodes along a routing path to thefinal destination computing node, the substrate network address that isused may correspond to a first such intermediate computing node as aninitial destination. After block 550, the routine in block 555 thenrewrites the communication header in accordance with a networkingaddress protocol for one or more intermediate networks between thecommunication manager module and the destination computing node usingthe physical substrate network address retrieved in block 550. Theheader re-writing may further include changing other information in thenew header, including changing a virtual network address for the sendingcomputing node to be a corresponding physical substrate network addressso as to direct responses to a communication manager module associatedwith the sending computing node (e.g., to an edge communication managermodule for an external computing system that sent the communication),and in at least some embodiments includes modifying the receivedcommunication without encapsulation as part of an overlay of the virtualcomputer network over the substrate one or more intermediate physicalnetworks. Furthermore, for a communication whose destination hardwareaddress does correspond to a virtual networking device, the routine inblock 555 may further perform other modifications that correspond toproviding virtual networking functionality to emulate the actions andfunctionality that would be performed by the one or more virtualnetworking devices that would be used to forward the communication tothe destination computing node in accordance with the configured networktopology for the virtual computer network.

In block 560, the routine then facilitates providing of the modifiedoutgoing communication along the routing path to the destinationcomputing node, such as by initiating forwarding of the modifiedoutgoing communication over the substrate intermediate network(s) to thedestination computing node. While not illustrated here, in otherembodiments various additional types of processing may be performed foroutgoing node communications, such as to verify that the communicationsare valid or otherwise authorized in various ways (e.g., to verify thatthe sending computing node is authorized to send communications to thedestination computing node, such as based on being associated with thesame entity or part of the same virtual computer network, based on thesending and destination computing nodes being associated with differententities that are authorized to inter-communicate, based on the type ofcommunication or other information specific to the communication, etc.),to add and/or remove a VLAN identifier or other information about a VLANconfigured for the network topology and associated with thecommunication (e.g., based on the type of VLAN communication linksassociated with the sending and destination computing nodes, such as ifthe communication manager module for an outgoing communication performssuch a modification rather than the communication manager module for anincoming communication), etc.

If it is instead determined in block 547 that the received communicationis a routing-related communication intended for a virtual networkingdevice, the routine continues instead to block 581 to forward thereceived routing-related communication to the Network Routing Managerroutine for analysis, as discussed in greater detail with respect toFIGS. 6A-6B. The communication may be forwarded to the Network RoutingManager routine in various manners, including by re-headering thecommunication and forwarding it over the intermediate substrate networkto a location of the Network Routing Manager module that performs theNetwork Routing Manager routine, in a manner similar to that discussedin greater detail elsewhere. In other embodiments, the routine mayinstead extract particular routing information from the communicationand instead forward the extracted routing information to the NetworkRouting Manager routine as part of such a communication or instead inanother manner (e.g., via a defined API of the Network Routing Manager).After block 581, the routine continues to block 583 to optionallygenerate a response communication to the computing node or externaldevice from which the routing-related communication was received (e.g.,to acknowledge receipt), such as if the predefined routing protocol inuse includes such response communications, and forwards the generatedresponse communication to that sender computing node or external device.After block 583, the routine continues to block 595.

If it is instead determined in block 510 that the received message is anincoming node communication for one of the managed computing nodes, suchas an inter-node communication from an external computing node that isnot managed by the communication manager module and/or a routing-relatedcommunication from the Network Routing Manager routine (e.g., arouting-related communication that is spoofed to include senderinformation corresponding to a virtual networking device), the routinecontinues to block 565. If the routine is being performed for acommunication manager module that operates as an edge module to manageconnections between a group of computing nodes connected by a substratenetwork (e.g., at a data center) and external computing systems, suchincoming communications correspond to communications exiting the groupof inter-connected computing nodes to external computing systems. Inblock 565, the routine identifies the physical substrate networkaddresses for the sender and for the destination computing node from thecommunication header. After block 565, the routine continues to block570 to optionally verify that the incoming communication is valid in oneor more ways. For example, the routine may determine whether thephysical substrate network address for the sender is actually mapped toa sending computing node that corresponds to the source physicalsubstrate network address location, such as based on interactions with asystem manager module and/or based on other information previouslyobtained and stored by the routine. In addition, the routine maydetermine whether the physical substrate network address for thedestination computing node corresponds to an actual managed computingnode, or to an external computing system or device with whichcommunications are authorized. While not illustrated here, if anincoming communication is determined to not be valid, the routine maytake various actions not shown, such as to generate one or more errorsand perform associated processing and/or drop the incoming communicationwithout forwarding it to the indicated destination node. For example, ifthe incoming communication indicates a destination network address thatdoes not correspond to a current managed computing node or authorizedexternal device, the routine may drop the incoming communication and/orinitiate an error message, although in some embodiments such errormessages are not sent to the sending computing node.

In the illustrated embodiment, after block 570, the routine continues toblock 575 to retrieve the hardware address and the network address thatare mapped to the physical destination substrate network address, and torewrite the communication header in a manner corresponding to thevirtual computer network, so that it includes information directed tothe destination computing node. For example, in some embodiments thedestination network address may be obtained from the destinationphysical substrate network address itself, such as from a subset of thebits of the destination physical substrate network address, while inother embodiments the destination network address may be indicated inother manners (e.g., stored in one or more header fields of thecommunication). In addition, the destination hardware address may havepreviously been mapped to information about the location of thedestination computing node, such as previously discussed with respect toblock 525. In situations in which such prior mapping has not occurred,the routine may instead perform blocks 515-525 to obtain suchinformation, although those actions are not illustrated here. Theroutine may similarly rewrite the communication header for the virtualcomputer network so that it appears to be sent from a sender with asource virtual network address and source hardware address correspondingto the sender. In addition, in at least some embodiments, the routine inblock 575 may further perform other modifications to the incomingcommunication that correspond to providing virtual networkingfunctionality to emulate the actions and functionality that would beperformed if the communication was forwarded in a manner associated witha configured VLAN for the managed virtual computer network, such as toadd and/or remove a VLAN identifier or other VLAN information, althoughin other embodiments such a modification is not performed for anincoming communication if it was instead performed for the communicationwhen outgoing for another communication manager module, or if nomodification is needed based on the configured VLAN communication linksassociated with the sending and destination computing nodes.Furthermore, in at least some embodiments, the routine in block 575 mayfurther perform other modifications to the incoming communication thatcorrespond to providing virtual networking functionality to emulate theactions and functionality that would be performed by one or more virtualnetworking devices that would have been used to forward thecommunication to the destination computing node in accordance with theconfigured network topology for the virtual computer network. Afterblock 575, the routine continues to block 580 to facilitate providing ofthe modified incoming communication to the destination computing node,such as by initiating forwarding of the modified incoming communicationto the destination node. While the incoming communication may be arouting-related communication that is directed to a managed computingnode of the communication manager module by the Network Routing Manager,such routing-related communications may in other embodiments instead begenerated by the communication manager module that manages thatcomputing node (e.g., as instructed by the Network Routing Manager, andsuch as to include routing information supplied by the Network RoutingManager), and provided to the destination computing node.

If it is instead determined in block 510 that a message of another typeis received, the routine continues to block 590 to perform anotherindicated operation as appropriate, such as to store information aboutentities associated with particular computing nodes, store informationabout configured network topologies for particular virtual computernetworks (including information about computing nodes that areassociated with target virtual network addresses for at least somesending computing nodes), respond to requests and other messages fromcomputing nodes in a manner to provide virtual networking functionalitycorresponding to configured network topologies for virtual computernetworks (e.g., by emulating actions and other functionalities thatwould be performed by specified virtual networking devices if they werephysically implemented), update previously mapped or stored informationto reflect changes with respect to computing nodes that are beingmanaged or to remote computing nodes, etc. The storing and/or updatingof stored information may be initiated in various manners, such as byreceiving information in response to previous requests, receivinginformation that is proactively pushed to the routine without acorresponding request, etc. In addition, in embodiments in which clientspay fees for various types of functionality provided to them, such asusage-based fees, the routine may further track corresponding usage andprovide such information to the system manager module (e.g.,periodically, as soon as usage occurs, etc.).

After blocks 560, 580, 583, or 590, or if it is instead determined inblock 530 that the processing is not being performed with respect to anoutgoing communication, the routine continues to block 595 to determinewhether to continue, such as until an explicit indication to terminateis received. If it is determined to continue, the routine returns toblock 505, and if not continues to block 599 and ends.

FIGS. 6A-6B are a flow diagram of an example embodiment of a CNS NetworkRouting Manager routine 600. The routine may be provided by, forexample, execution of the NRM module 342 of FIG. 3, the NRM module 295of FIG. 2E, the Network Routing Manager module 170 of FIG. 1B, a NetworkRouting Manager module (not shown) of the CNS service 105 of FIG. 1A,and/or a Network Routing manager module (not shown) that operates inconjunction with the System Manager module 290 of FIGS. 2A-2D, such asto manage the configuration of specified network topology informationfor managed computer networks, including the use of target virtualnetwork addresses and associated computing nodes. In the illustratedembodiment, the routine may be invoked by, for example, execution ofblock 480 of FIG. 4 and/or directly in response to a request initiatedby a client of the CNS system, and may receive information forwarded byblock 581 of FIGS. 5A-5B. In addition, in the illustrated embodiment ofthe routine, a Network Routing manager module facilitates theconfiguration of specified network topology information for managedvirtual computer networks, and provides corresponding information to asystem manager module that further interacts with communication managermodules based on the specified network topology information—in otherembodiments, however, various functionality may be distributed in othermanners, such as to combine some or all functionality of a NetworkRouting manager module, a system manager module, and/or one or morecommunication manager modules.

The routine begins at block 605, where an indication is received ofinformation related to the configuration of a managed computer network.The routine continues to block 610 to determine whether the indicationis related to the initial configuration of network topology informationfor a managed virtual computer network, such as part of initiallycreating the managed virtual computer network. If so, the routinecontinues to block 615 to obtain various information about theconfigured network topology, such as information previously received inblock 605, or by interacting with a user or other entity for a clientproviding the configuration information. The obtained information mayinclude, for example, information regarding logical subnets of themanaged computer network and optionally corresponding computing nodesand/or virtual networking devices, information about any computing nodesthat are configured to act as intermediate computing nodes for at leastsome inter-node communications for at least some computing nodes,information about any special-purpose handling of at least someinter-node communications for at least some computing nodes, etc. Insome situations, the configuration information may indicate one or moretarget virtual network addresses and associated configurationinformation, such as a type of use of the target virtual network address(e.g., as an anycast target virtual network address that may havemultiple associated computing nodes; as a target virtual network addressthat may be serially assigned to multiple distinct computing nodes, suchas computing nodes that claim ownership of the target virtual networkaddress; etc.). Furthermore, in some embodiments, the configurationinformation may indicate particular computing nodes that are allowed tobe used and/or not used with particular target virtual networkaddresses.

The routine next continues to blocks 620-635 to analyze the receivedconfiguration information to determine particular information to use inrouting communications between computing nodes. In particular, in block620, the routine selects the next computing node of the managed computernetwork, beginning with the first. In block 625, the routine then, foreach other computing node of the managed computer network, determines ifinter-node communications from the selected computing node to the othercomputing node are configured to pass through one or more intermediatecomputing nodes based on the configured network topology. If multiplecomputing nodes are associated with a particular target virtual networkaddress, such as an anycast target virtual network address, theillustrated routine may select a particular one of those multipleassociated computing nodes that will be used for the anycast targetvirtual network address for the selected computing node (such that otherof those multiple associated computing nodes may not be directlyreachable by the selected computing node), although in other embodimentsthe routine may defer such a selection until a later time (e.g., a timethat the selected computing node uses that targeted virtual networkaddress) or later modify an initial selection based on additionalinformation (e.g., routing information that is received during operationof the managed computer network that corresponds to the target virtualnetwork address). If the inter-node communications are determined topass through a series of one or more intermediate computing nodes, theroutine selects the substrate network address for a first suchintermediate computing node to use for communications directed from theselected computing node to the other computing node, and otherwiseselects the substrate network address for the other computing node touse for such inter-node communications.

After block 625, the routine continues to block 630 and stores thedetermined information (e.g., in a manner accessible to a system managermodule, such as by providing the information to the system managermodule), and optionally provides the configuration information (e.g., ina push manner) to one or more communication manager modules, includingthe communication manager module that manages the selected computingnode. In block 635, the routine then determines if there are morecomputing nodes to analyze, and if so returns to block 620, andotherwise continues to block 695. While illustrated here as performingthe analysis in a serial manner for individual computing nodes, in otherembodiments and situations such analysis may be performed in othermanners.

If it is instead determined in block 610 that the received indication isrelated to a possible update of network topology information for amanaged virtual computer network, such as based on a routingcommunication received via a communication manager module and/or basedon an update to network topology information specified by the associatedclient, the routine continues to block 645. In block 645, the routinedetermines whether routing information has been received in a routingcommunication for a managed computer network, and if so continues toblock 655 to extract the routing information from the communication andoptionally store it for later use (e.g., as discussed with respect toblock 690). After block 655, the routine continues to block 660 toanalyze the received routing information to determine if the routinginformation includes any indications of external public networkaddresses to be associated with the managed computer network, such thatexternal computer systems that send communications to those publicnetwork addresses will be routed to the computing node that sent therouting information. If so, the routine in block 660 optionally verifiesthe indicated external public network addresses as being authorized foruse, and initiates an external announcement of those external publicnetwork addresses, as discussed in greater detail elsewhere. After block660, the routine continues to block 665 to analyze the received routinginformation to determine if the computing node that sent the routinginformation has newly asserted availability of a particular targetvirtual network address via that computing node, or if a differentcomputing node has become associated with a target virtual networkaddress based on a previously associated computing node becomingunavailable or otherwise no longer maintaining that association. It willbe appreciated that in other embodiments various of the functionalitymay be performed in other manners, such as to perform the analysis ofblocks 660 and 665 at the same time.

If it is instead determined in block 645 that the network topologyinformation is received from the client, the routine continues to block650 to update the stored network topology information for the managedvirtual computer network in accordance with the received information.After blocks 650 or 665, the routine continues to block 667 to determineif there are any computing nodes of the managed computer network toupdate with the received information, such as based on the configurednetwork topology of the managed computer network. In particular, if thereceived information was directed to a particular virtual networkingdevice of the managed computer network, the routine may identify othercomputing nodes of the managed computer network to which that networkingdevice would forward routing information if that networking device werephysically provided, such as based on the network topology of themanaged computer network. For example, if a routing communication wassent by a first computing node of the managed computer network thatparticipates in one or more routing-related protocols, and was directedto a specified virtual networking device of the managed computer networkthat connects the first computing node to one or more other secondcomputing nodes of the managed computer network that also participate inone or more routing-related protocols for the managed computer network,the routine may select at least one (e.g., all) of the second computingnodes to receive corresponding routing information. The routine maydetermine such computing nodes of the managed computer network thatparticipate in one or more routing protocols for the managed computernetwork in various manners, such as based on configuration informationsupplied by a client for whom the managed computer network is provided,and/or based on tracking which computing nodes issue routing-relatedcommunications. After block 667, the routine then continues to block 669to provide to the determined computing nodes corresponding routinginformation (e.g., routing information obtained in block 655), such asby sending a routing communication with such corresponding routinginformation to each of the determined computing nodes. In particular, ifrouting information obtained in block 655 was directed to a particularvirtual networking device of the managed computer network, the routinemay determine particular routing information that the particular virtualnetworking device would forward on if that networking device werephysically provided, such as in a manner relative to a position of thatvirtual networking device based on the network topology of the managedcomputer network. For example, if the virtual networking device isconfigured to select shortest paths to destinations by usingdistance-based cost information, the routine may use its availableinformation for the entire managed computer network to determine thevarious shortest paths from the location of the virtual networkingdevice to the various possible destinations, including based on routinginformation received in block 655. It will be appreciated that theroutine may operate in other manners in other embodiments, such as bydetermining best path information in manners other than usingdistance-based cost information, by determining best path informationfor a particular virtual networking device in a localized manner forthat virtual networking device rather than using globally availableinformation for the entire managed virtual computer network, etc.

After block 669, the routine continues to block 678 to determine if theupdates from the received information affect the routing paths for anyinter-node communications, such as in a manner similar to thatpreviously discussed with respect to blocks 620-635. After block 678,the routine continues to block 680 to determine if any routing paths forinter-node communications are affected by the information received inblock 605, such as based on the determination with respect to block 678.If so, the routine continues to block 682, and otherwise continues toblock 684. In block 682, the routine determines a new destinationsubstrate network address to use for each destination computing node inan affected routing path for inter-node communications for one or moresending computing nodes, in a manner similar to that discussed withrespect to block 625 and elsewhere. As discussed with respect to blocks620-630, in some embodiments, the determination of a destinationsubstrate network address to use for a destination computing node mayinclude selecting one of multiple associated computing nodes for atarget virtual network address and/or may include maintaining possibleassociations of multiple such destination substrate network addressesfor a target virtual network address that correspond to multiple suchassociated computing nodes. The routine then stores the determined newinformation as part of the network topology information for the managedcomputer network. After block 682, or if the routine determines in block680 that there are not any affected paths, the routine continues toblock 684 to optionally update the mapping information stored by one ormore communication manager modules affected by changes that result fromthe information received in block 605, such as one or more routing pathchanges. In addition, in block 684, the routine optionally initiates thesending of one or more routing-related communications to the managedcomputer network for which the information received in block 605corresponds (e.g., to computing nodes of the managed computer networkthat participate in a routing protocol for the managed computernetwork), such as to reflect routing changes.

If it is instead determined in block 610 that a message of another typeis received, the routine continues to block 690 to perform anotherindicated operation as appropriate. For example, as discussed in greaterdetail elsewhere, in some embodiments the routine may generate and sendrouting information to at least some computing nodes of a managedcomputer network that participate in a routing protocol for that managedcomputer network, including in a manner that is specific to the contextof the recipient computing node within the configured network topologyof the managed computer network, and/or in a manner to coordinate therouting information provided to the various computing nodes. Thegeneration and sending of such routing information may be initiated invarious manners, such as based on received routing information and/orother updates to configured network topology information, in a periodicmanner, etc. In addition, in embodiments in which clients pay fees forvarious types of functionality provided to them, such as fees forparticular uses of substrate networks and/or for at least somerouting-related communications that are received or sent, the routinemay further track corresponding operations and provide such informationto the system manager module (e.g., periodically, as soon as usageoccurs, etc.).

After blocks 689 or 690, or if it is determined in block 635 that thereare no more computing nodes, the routine continues to block 695 todetermine whether to continue, such as until an explicit indication toterminate is received. If it is determined to continue, the routinereturns to block 605, and if not continues to block 699 and ends.

In addition, various embodiments may provide mechanisms for customer orother client users and other entities to interact with an embodiment ofthe system manager module for purpose of configuring computing nodes andtheir communications. For example, some embodiments may provide aninteractive console (e.g. a client application program providing aninteractive user interface, a Web browser-based interface, etc.) fromwhich users can manage the creation or deletion of virtual computernetworks, and the configuration of network topology information forvirtual computer networks (including target virtual network addressesand associated computing nodes), as well as more general administrativefunctions related to the operation and management of hosted applications(e.g., the creation or modification of user accounts; the provision ofnew applications; the initiation, termination, or monitoring of hostedapplications; the assignment of applications to groups; the reservationof time or other system resources; etc.). In some embodiments, some orall of the functionality of an embodiment of the CNS system may beprovided in exchange for fees from users or other entities acting ascustomers or other clients of the CNS system, as discussed previously,and if so the mechanisms for such clients to interact with an embodimentof the system manager module may include mechanisms for users and otherentities to provide payment and payment-related information, as well asto monitor corresponding payment information, and the CNS system maytake various actions to obtain appropriate payments from such clients.In addition, some embodiments may provide an API that allows othercomputing systems and programs to programmatically invoke at least someof the described functionality, such as APIs provided by libraries orclass interfaces (e.g., to be invoked by programs written in C, C++, orJava) or otherwise, and/or using network service protocols such as viaWeb services. Additional details related to the operation of exampleembodiments of a program execution service and/or configurable networkservice with which the described techniques may be used are available inU.S. application Ser. No. 11/394,595, filed Mar. 31, 2006 and entitled“Managing Communications Between Computing Nodes;” U.S. application Ser.No. 11/395,463, filed Mar. 31, 2006 and entitled “Managing Execution ofPrograms by Multiple Computing Systems;” U.S. application Ser. No.11/692,038, filed Mar. 27, 2007 and entitled “ConfiguringIntercommunications Between Computing Nodes;” and U.S. application Ser.No. 12/332,214, filed Dec. 10, 2008 and entitled “Providing Access ToConfigurable Private Computer Networks;” each of which is incorporatedherein by reference in its entirety. In addition, additional detailsrelated to the management of provided virtual networks that may be usedby at least some embodiments of a CNS system, such as in conjunctionwith an Overlay Network Manager module of such a CNS system, areavailable in U.S. application Ser. No. 12/060,074, filed Mar. 31, 2008and entitled “Configuring Communications Between Computing Nodes;” inU.S. application Ser. No. 12/414,260, filed Mar. 30, 2009 and entitled“Providing Logical Networking Functionality For Managed ComputerNetworks;” and in U.S. application Ser. No. 12/491,818, filed Jun. 25,2009 and entitled “Providing Virtual Networking Functionality ForManaged Computer Networks;” each of which is also incorporated herein byreference in its entirety.

It will also be appreciated that, although in some embodiments thedescribed techniques are employed in the context of a data centerhousing multiple physical machines hosting virtual machines, otherimplementation scenarios are also possible. For example, the describedtechniques may be employed in the context an organization-wide networkor networks operated by a business or other institution (e.g. auniversity) for the benefit of its employees and/or members.Alternatively, the described techniques could be employed by a networkservice provider to improve network security, availability, andisolation. In addition, example embodiments may be employed within adata center or other context for a variety of purposes. For example,data center operators or users that sell access to hosted applicationsto customers may in some embodiments use the described techniques toprovide network isolation between their customers' applications anddata; software development teams may in some embodiments use thedescribed techniques to provide network isolation between variousenvironments that they use (e.g., development, build, test, deployment,production, etc.); organizations may in some embodiments use thedescribed techniques to isolate the computing resources utilized by onepersonnel group or department (e.g., human resources) from the computingresources utilized by another personnel group or department (e.g.,accounting); or data center operators or users that are deploying amulti-component application (e.g., a multi-tiered business application)may in some embodiments use the described techniques to providefunctional decomposition and/or isolation for the various componenttypes (e.g., Web front-ends, database servers, business rules engines,etc.). More generally, the described techniques may be used tovirtualize physical networks to reflect almost any situation that wouldconventionally necessitate physical partitioning of distinct computingsystems and/or networks.

It will also be appreciated that in some embodiments the functionalityprovided by the routines discussed above may be provided in alternativeways, such as being split among more routines or consolidated into fewerroutines. Similarly, in some embodiments illustrated routines mayprovide more or less functionality than is described, such as when otherillustrated routines instead lack or include such functionalityrespectively, or when the amount of functionality that is provided isaltered. In addition, while various operations may be illustrated asbeing performed in a particular manner (e.g., in serial or in parallel)and/or in a particular order, those skilled in the art will appreciatethat in other embodiments the operations may be performed in otherorders and in other manners. Those skilled in the art will alsoappreciate that the data structures discussed above may be structured indifferent manners, such as by having a single data structure split intomultiple data structures or by having multiple data structuresconsolidated into a single data structure. Similarly, in someembodiments illustrated data structures may store more or lessinformation than is described, such as when other illustrated datastructures instead lack or include such information respectively, orwhen the amount or types of information that is stored is altered.

From the foregoing it will be appreciated that, although specificembodiments have been described herein for purposes of illustration,various modifications may be made without deviating from the spirit andscope of the invention. Accordingly, the invention is not limited exceptas by the appended claims and the elements recited therein. In addition,while certain aspects of the invention are presented below in certainclaim forms, the inventors contemplate the various aspects of theinvention in any available claim form. For example, while only someaspects of the invention may currently be recited as being embodied in acomputer-readable medium, other aspects may likewise be so embodied.

What is claimed is:
 1. A computer-implemented method comprising:instantiating, by one or more computing systems of a configurablenetwork service, a virtual computer network for a client based onconfiguration information received from the client, wherein theconfiguration information indicates a first subnet and a second subnetthat interconnect multiple computing nodes of the virtual computernetwork; intercepting, by the one or more computing systems and whilethe virtual computer network is in use, a gratuitous address resolutionprotocol (GARP) communication that includes information indicating afirst computing node of the multiple computing nodes is using a firstvirtual network address for the virtual computer network; updating, bythe one or more computing systems and based on the information includedin the GARP communication, the configuration information for the virtualcomputer network to associate the first virtual network address with thefirst computing node; and sending, by the one or more computing systemsand based at least in part on the updated configuration information, acommunication directed to the first virtual network address to the firstcomputing node.
 2. The computer-implemented method of claim 1 whereinthe instantiating of the virtual computer network includes: receiving,by the one or more computing systems, a request from the client tocreate the virtual computer network; selecting, by the one or morecomputing systems and for use with the virtual computer network, asubset of a plurality of computing nodes that are provided by theconfigurable network service for use by a plurality of clients of theconfigurable network service; and creating, by the one or more computingsystems, the virtual computer network for the client using the computingnodes of the selected subset.
 3. The computer-implemented method ofclaim 1 further comprising receiving the configuration informationbefore the instantiating of the virtual computer network, wherein theconfiguration information specifies a configured manner for using thefirst virtual network address that includes associating the firstvirtual network address with a second computing node of the multiplecomputing nodes, and wherein the instantiating of the virtual computernetwork further includes, before the intercepting of the GARPcommunication, assigning multiple virtual network addresses for thevirtual computer network to at least some of the multiple computingnodes, including associating the first virtual network address with thesecond computing node based on the configuration information.
 4. Thecomputer-implemented method of claim 1 wherein the configurationinformation further indicates that a virtual router device of thevirtual computer network interconnects the first and second subnets,wherein the GARP communication is directed to the virtual router device,and wherein the method further comprises, before the intercepting of theGARP communication, assigning the first virtual network address to asecond computing node, and forwarding to the second computing node oneor more initial communications directed to the first virtual networkaddress, wherein the forwarding of the one or more initialcommunications includes emulating functionality of the virtual routerdevice for the forwarding.
 5. The computer-implemented method of claim 1wherein the configuration information further indicates that the firstvirtual network address is allowed to be associated with at least thefirst computing node, and wherein the updating of the configurationinformation includes determining, by the one or more computing systems,that the first computing node is authorized to be associated with thefirst virtual network address based at least in part on theconfiguration information.
 6. The computer-implemented method of claim 5wherein the first virtual network address is configured to be an anycastaddress for the virtual computer network that is associated with a groupof the multiple computing nodes, and wherein the updating of theconfiguration information includes designating that the first computingnode is part of the group.
 7. The computer-implemented method of claim 6wherein a second computing node is also in the group, and wherein themethod further comprises, after the updating of the configurationinformation, forwarding one or more additional communications directedto the first virtual network address to the second computing node. 8.The computer-implemented method of claim 5 wherein the first virtualnetwork address is configured to be migratable between and seriallyassociated with two or more of the multiple computing nodes, and whereinthe updating of the configuration information for the first virtualcomputer network includes transferring an association of the firstvirtual network address from a second computing node to the firstcomputing node.
 9. The computer-implemented method of claim 5 whereinthe first virtual network address is not associated with any of themultiple computing nodes before the updating, wherein the configurationinformation further indicates that a virtual router device of thevirtual computer network interconnects the first and second subnets, andwherein the GARP communication is directed to the virtual router deviceby the first computing node to newly claim the first virtual networkaddress.
 10. The computer-implemented method of claim 1 furthercomprising receiving a second GARP communication with informationfurther indicating that a third of the multiple computing nodes is newlyassociated with a second virtual network address, and determining, bythe one or more computing systems, to prevent the third computing nodefrom being associated with the second virtual network address based atleast in part on the configuration information.
 11. Thecomputer-implemented method of claim 1 wherein the configurationinformation further indicates that a virtual router device of thevirtual computer network interconnects the first and second subnets,wherein the instantiating of the virtual computer network includesemulating functionality of the virtual router device without using aphysical device in an underlying substrate network on which the virtualcomputer network is overlaid to provide routing operations of thevirtual router device for communications of the virtual computernetwork, and wherein the intercepting of the GARP communication includesreceiving the GARP communication from a sender and inhibiting forwardingof the GARP communication over the substrate network.
 12. A system,comprising: one or more hardware processors; and one or more memorieswith stored instructions that, when executed by at least one of the oneor more hardware processors, cause the system to perform automatedoperations including at least: providing a virtual computer network fora client based on configuration information received from the clientthat specifies first and second subnets of the virtual computer networkinterconnecting multiple computing nodes of the virtual computernetwork; receiving, while the virtual computer network is in use andafter one or more communications have been sent to a first virtualaddress of the virtual computer network, a gratuitous address resolutionprotocol (GARP) communication that is from a first computing node of themultiple computing nodes and that includes information indicating thefirst computing node newly claims use of the first virtual networkaddress in the virtual computer network; and forwarding, based at leastin part on the information included in the GARP communication, anadditional communication directed to the first virtual network addressto the first computing node.
 13. The system of claim 12 wherein theproviding of the virtual computer network includes, before the receivingof the GARP communication, assigning multiple virtual network addressesfor the virtual computer network to at least some of the multiplecomputing nodes, the assigning including associating the first virtualnetwork address with a second computing node of the multiple computingnodes, and wherein the stored instructions include software instructionsthat further cause the system to update, based on the GARPcommunication, the configuration information for the virtual computernetwork to associate the first virtual network address with the firstcomputing node.
 14. The system of claim 13 wherein the storedinstructions further cause the system to implement at least somefunctionality of a configurable network service that provides virtualcomputer networks to multiple clients of the configurable networkservice, and wherein the providing of the virtual computer network forthe client further includes: receiving a request from the client tocreate the virtual computer network and to provide a virtual routerdevice in the virtual computer network between the first and secondsubnets; selecting a subset of a plurality of computing nodes that areprovided by the configurable network service for use by the multipleclients; and implementing the virtual computer network for the clientusing the computing nodes of the selected subset, including emulatingfunctionality of the virtual router device as part of forwardingcommunications between the first and second subnets, and wherein theGARP communication is directed from the first computing node to thevirtual router device and is intercepted by the configurable networkservice as part of further emulating of the functionality of the virtualrouter device.
 15. A non-transitory computer-readable storage mediumhaving stored contents that cause a computing system to performautomated operations including at least: providing, by the computingsystem, and based on configuration information received from a clientfor a virtual computer network having first and second subnets thatinterconnect multiple computing nodes of the virtual computer network,the virtual computer network for the client; receiving, by the computingsystem, a gratuitous address resolution protocol (GARP) communicationthat includes information indicating use of an external public networkaddress within the virtual computer network by a first computing node ofthe multiple computing nodes; initiating a routing announcement externalto the virtual computer network indicating use of the external publicnetwork address by the virtual computer network, to cause the externalpublic network address to be identifiable by computer systems externalto the virtual computer network as available for use in sendingcommunications to the virtual computer network; and receiving, by thecomputing system and after initiating the routing announcement, anexternal communication directed to the external public network addressfrom a sending computer system external to the virtual computer network,and forwarding the external communication to the first computing node.16. The non-transitory computer-readable storage medium of claim 15wherein the GARP communication is directed to the virtual router deviceby the first computing node while the virtual computer network is in useto newly claim use of the external public network address within thevirtual computer network, and wherein the stored contents includesoftware instructions that, when executed by the computing system,further cause the computing system to update, based on the GARPcommunication, the configuration information for the virtual computernetwork to associate the external public network address with the firstcomputing node.
 17. The non-transitory computer-readable storage mediumof claim 15 wherein the computing system is part of a configurablenetwork service that provides multiple virtual computer networks tomultiple remote clients, wherein the configurable network serviceprovides a plurality of computing nodes for use with the multiplevirtual computer networks and uses a subset of the plurality to providethe virtual computer network for the client, and wherein the initiatedrouting announcement is provided to devices external to the configurablenetwork service.
 18. The non-transitory computer-readable storage mediumof claim 15 wherein the initiating of the routing announcement includessending, by the computing system, information about the external publicnetwork address to one or more computing devices of the client externalto the virtual computer network.
 19. The non-transitorycomputer-readable storage medium of claim 15 wherein the computingsystem is part of a configurable network service that provides thevirtual computer network, wherein the external public network address isassigned to the client by one or more entities external to theconfigurable network service, and wherein the stored contents furthercause the computing system to, before initiating the routingannouncement, verify that the client is authorized to use the externalpublic network address outside the configurable network service.
 20. Thenon-transitory computer-readable storage medium of claim 15 wherein thestored contents further cause the computing system to receive anadditional GARP communication that includes information indicating anadditional external public network address for use with the virtualcomputer network, and to determine, based at least in part on theconfiguration information, to prevent initiation of any routingannouncements external to the virtual computer network for theadditional external public network address.